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ABSTRACT 



The Massachusetts Workplace Literacy Consortium was funded 
through a 3-year National Workplace Literacy Program grant. During the second 
grant year, the consortium operated 25 program sites that were almost evenly 
divided between health care and manufacturing. The following were the primary 
reasons programs were started: reduce error and waste (76%) , accommodate 
change in work organization or process (76%) , and make greater use of 
employees who are nonnative speakers of English (68%) . Two outside evaluators 
were hired to evaluate the programs during their second year. The evaluation 
focused on the following: extent to which the consortium goals were being 
met; level of implementation of the Massachusetts Indicators of Quality for 
Workplace Literacy Programs and the indicators' relationship to learner 
outcomes, workplace outcomes, arid program partnership; relationship between 
instructional methodologies and worker and workplace outcomes; and ways 
programs could improve. The evaluation documented some learning gains, 
particularly in English, math, problem solving, and reasoning. (Twenty- five 
tables are included. Appendixes constituting approximately 50% of this 
document contain the following: indicators of quality; Indicators PLUS 
protocol for program coordinators; quality indicators' results; self- score 
sheet; cost figures; Department of Education data collection forms; plan of 
operation; and revised guidelines for curriculum documents.) (MN) 
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Working Toward Quality 
Executive Summary 

The Massachusetts Workplace Literacy Consortium (the Consortium) was funded for three 
years from November 1, 1994 to October 31, 1997 through the National Workplace Literacy 
Program (NWLP). The Consortium is administered through the Adult and Community Learning 
Services Cluster (ACLS) of the Massachusetts Department of Education by ACLS/Consortium 
staff. This report summarizes the evaluation activities conducted in the second grant year and 
presents their results. Quantitative data are reported on Period 3 only, the six months from 
November 1, 1996 to April 30, 1997. The report uses four evaluation objectives as a framework 
for reporting results and assessing the Consortium’s strengths and challenges. Before presenting 
conclusions for each objective, we present a summary of descriptive findings. 

There are 25 program sites almost evenly divided between health care and manufacturing; 
one is higher education. Eleven sites are small businesses (less than 250); three are inCdium (250- 
500); nine are large. Slightly less than half are unionized. The three primary reasons for starting a 
program are to: reduce error and waste (76%); accommodate a change in work organization or 
process (76%); and make greater use of employees who are (ESOL) English Speakers of Other 
Languages (68%). Seven sites (28%) offer complete paid release time as an incentive and 17 
(68%) offer partial paid release time. The students are mostly white middle aged adults (just over 
40 years of age). Over 80% are bom outside the US, with a slight majority being female. The 
students generally have less than a high school education but more than elementary school. They 
have enjoyed stable employment for an average of eight years. 

The overall average cost per site is $35,939, of which $14,908 is from the NWLP grant 
and the remainder is from company cash contributions (largely, if not exclusively, paid release time 
for instruction) and in-kind contributions of the education or business partners or both. The 
programs cost about $875 per student during Period 3, including $348 of public funds and $527 of 
non-public funds. The cost per student contact hour is about $14.23, with a little less than half that 
from the NWLP funds and over half from the company's cash and other in-kind contributions. 

There were 138 courses offered during Period 3 across all sites, with an average of 
between five and six per site. The range was wide. One site offered only one course; m.ost sites 
offered from two to eight courses; and two offered significantly larger numbers, 16 and 31 
courses. The numbers of students in each class averaged a little over nine, with a fairly narrow 
distribution around that figure. The courses also varied quite a bit in length, with an average of 
over 65 hours per course; the distribution was quite even within the extremes of 24 to 120 hours. 

The primary types of content taught are ESOL, mostly at the intermediate level, followed 
by the beginner level. A substantial amount, 16 percent, of pre-GED preparation is also taught. 
The most predominant emphasis within the courses is communications, in 38 percent of the 
courses. This is followed by reading and writing combined, then reading, writing and literacy 
each as separately reported emphases. Placement and assessment tools used most frequently are 
student interviews, portfolios, Individual Education Plans (EEPs), and "other" methods. Courses 
with primary emphasis on reading, math, or reading and math are offered in substantially greater 
proportion at manufacturing than at health sites. Courses with primary emphasis on writing, 
communications, or reading and writing have a substantially higher proportion offered in health 
than in manufacturing sites. 

Results indicate some learning gains, particularly in writing English, and also in speaking, 
understanding and reading English, using math, and solving problems and using reasoning. The 
gains in English usage stand to reason since most of the courses addressed those content areas. 

The problem-solving and reasoning gains look as strong as the English gains, even though there 
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are no programs that reported course focus in that area. Perhaps problem solving and reasoning 
gains are component parts of gains in writing, speaking, understanding and reading. 



Evaluation Objectives and Conclusions: 

Objective #1. Determine if the goals of the Consortium are being met 

Overall the Consortium is meeting its goals: 

• The Consortium is serving the population it intended to serve. The Consortium has met its goal 
to serve 1,200 workers per year. 

• Workers have achieved noticeable learning gains . It is a key finding of this evaluation that 
workers made noticeable learning gains in Period 3 alone. Workers were given a scale on which 
to rate their pre instruction status and their post instruction status on seven skills areas. Gains 
were particularly noted in writing English, but also to a fair degree in speaking, understanding 
and reading English, using math, and solving problems and using reasoning. A sum of gains of 
0.66 signifies that, overall, workers rated themselves higher at the end of Period 3 instruction on 
the seven skills areas than at the beginning. A 0.66 gain represents a 22% gain of the scale’s 
range. Thus the Consortium has made progress toward its goal of improving the literacy skills 
of workers. (The number of workers for whom there are data on learning gains is from 158 to 
181 out of the total 1,294 workers reported to have been served during Period 3.) 

A series of questions also asked workers for reports on whether other outcomes occurred. As 
with the self-ratings in the skill areas referenced above, these data are reported by a small 
proportion of the employees served during the period. Nonetheless, ninety-five percent of the 
respondents indicated that they learned what they wanted to, which speaks well for participant 
satisfaction with instruction and suggests the achievement of learning gains as well. 

• Some positive impact on the workplace has been achieved. Business partners and site personnel 
from 18 of 20 sites reporting state that there has been some positive impact of worker 
participation in a program on the workplace. The main areas of improvement are employee 
communications, participation in team work and work processes, and reduced scrap and rework. 
Fifteen of the 20 respondents (75 percent) cited improv^ communication as a workplace 
outcome, consistent with the Consortium’s ESOL demography, its primary course emphases and 
the types of services most often offered. 

• The Consortium is making wise use of federal dollars. The Consortium is using NWLP monies 
in the way that they were intended: to seed workplace education programs that have the potential 
for continuation beyond federal funding and to leverage monies and services from companies that 
support these important programs. 

• Local Planning and Evaluation Teams (PETsl are functioning as planned and may still beriefit 
from further technical assistance . Overall, local PETs are functioning as planned — as a hub of 
program activity and the embodiment of partnership at the local level. In Year 3 they might 
benefit from additional technical assistance and support from the Consortium Planning and 
Evaluation Team (CPET). 

• The CPET has ably guided the Consortium through a complex plan of service delivery to 25 
programs but would benefit from implementing a formal plannine and evaluation process to 
guide it throueh a final year . The CPET has served the Consortium well, largely in a monitoring 
role. ACLS/Consortium staff might now encourage CPET members to think more proactively 
about what they can accomplish in Year 3 through facilitating a formal planning and evaluation 
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process. The Consortium’s Year 3 emphasis on institutionalization suggests a need for 
additional focus on local evaluation activities and how they might serve institutionalization. 

• The Consortium has explored three types of distance learning opportunities and the results of 
their use are unclear at this time . The Consortium experienced “media initiation by fire’’ and 
emerged with reasonable products and significant learnings about how to proceed with future 
media projects. Actual results for students and programs are unclear at this time. It is clear, 
however, that it is not as easy to produce quality education products using telecommunications 
and computer media as it might seem. Multimedia learning will likely continue to grow rapidly 
as a field, and it is worthwhile for the Consortium to consider continuing production and 
dissemination of its own media products. Sufficient expertise, time and resources, however, 
need to be allocated to create worthwhile products and to evaluate their effects. 

• ACLS/Consortium staff learned that there must be a congmence between a new technology, 
instmctional content and the readiness of teachers and students to work with new technology . 
There was agreement among Coordinators and ACLS/Consortium staff that activities in all three 
media projects were somewhat premature and would have benefited from more development 
time. As noted below, at the end of Year 2, the Curriculum Working Group (CWG) is ready to 
share some of the materials and formats that it has been compiling. It took two years for the 
group to identify all the dimensions of its task and come to agreement about how to execute that 
task - documentation and dissemination of workplace curricula. Had the group been pressured 
to produce documentation in Year 1, it is unlikely that its format and content would be as rich as 
they are now. 

• The CWG is building the capacity of the Consortium at the same time that it is producing and 
disseminating valuable documentation of curricula . The CWG’s written products will likely be 
enormously useful to various audiences. The process the CWG used to document the curricula 
is also a product which may have many applications and a wide audience. The Massachusetts 
Department of Education and the Consortium and would be well-served if staff time were 
allocated to help with dissemination. 

» CPET members have been pursuing continuation of their programs but the level of business and 
union commitment to continue programs fas of May 19961 falls short of the Consortium 
standard . It would be advantageous for Coordinators to discuss and plan for program 
continuation more actively than before. ACLS/Consortium staff might troubleshoot any 
problems that Coordinators have with discussing or planning continuation by including the topic 
on CPET meeting agendas for the next several months. 



Objective #2: Determine the level of implementation of the Massachusetts 

Indicators of Quality for Workplace Literacy Programs and the relationship of the 
Indicators to learner outcomes, workplace outcomes and the program partnership 

• With modification the Quality Indicators can be useful for further research and evaluation. A 
central focus of the Consortium’s Year 2 Evaluation was testing the assumption that there is a 
relationship between the presence of Quality Indicators and positive outcomes at the learner, 
workplace, and partnership levels. The Quality Indicators intuitively have benefits when used as 
a guide for program development. The effort to test their validity by developing a scoring 
method for them and correlating those scores with outcomes is a useful research focus that we 
pursued this year but without a clear conclusion. There were several reasons for this: 

(1) The level of inference in the scoring process used by the evaluators is higher than some 
audiences are comfortable with. Efforts to create a scoring system based on performance 
standards for the Quality Indicators is underway and may eventually lead to better opportunities 
to study the relationship between them and outcomes. p 
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(2) Available outcome data was scant at best. Testing the relationship between Quality Indicators 
and outcomes relies as much on good outcome data as it does on a satisfactory scoring system 
for the Quality Indicators. Scant outcome data, a by-product of the unexpected dearth of data 
from the National Workplace Literacy Information System (NWLIS), as well as limited anecdotal 
reporting from local sites, hampered analysis of the Quality Indicators as much as any difficulty 
in the Quality Indicator scoring process. 

Continued work on the relationship between Quality Indicators and outcomes will likely prove 
useful to many audiences. Refined performance standards for the Quality Indicators can serve 
multiple purposes, including guiding programs in their development, possibly through a self- 
scoring process. Improved documentation of outcomes will also serve many audiences, among 
them local PETs who can use this information to argue for program continuation. The resources 
and time needed to continue the proper study of the relationship between Quality Indicators and 
outcomes, however, are at this time quite extensive. The DOE may be well served to continue 
this work internally with evaluator support and turn the focus of the Consortium evaluation onto 
questions of more immediate concern and impact, including how PETS are functioning. 

• The level of implementation of the Quality Indicators as scored in Year 2 is quite high. Given the 
limitations just stated, sites and partners have implemented the Quality Indicators at a very high 
level. The average ratings on 18 of the 25 Indicators (72 percent) were above 5 on a 6-point 
scale (with 6 as the highest level of implementation). The strongest areas were Staff, 

Curriculum, and Administration; the weakest was Assessment, Evaluation and Outcomes. 
Additionally, the level of Quality Indicator implementation was substantially higher for health 
sites than for manufacturing sites on 64 percent (16 of 25) of the Indicators, a result that is 
possibly explained by the fact that the partnerships generally work with either manufacturing 
sites or health sector sites exclusively. Since many activities covered in the Quality Indicators are 
mostly in the control of the education partner common to a number of business partners, then the 
extent to which the educational partner implements the Indicators will be reflected in the Quality 
Indicator scores of each of its constituent sites. It may be that those education partners with 
health sector business partners are more effective in implementing the Indicators. On the other 
hand, manufacturing companies may present more difficult barriers to implementing the "ideal" 
factors included in the Indicators, somewhat beyond the control of the educational partners, and 
these may be reflected in their systematically lower Quality Indicator ratings. 

• The relationship between Quality Indicator Implementation to Worker. Workplace, and 
Partnership Outcomes is not clear at this time . Correlations between self-reported learning gains 
and sites' Quality Indicator ratings were generally low. 

Objective #3: Determine the relationship between instructional methodologies and 

worker and workplace outcomes 

The initial emphasis in this objective was on the relationship of instructional characteristics to 
outcomes. We have added the relationship of business characteristics to outcomes. Conclusions 
are offered to stimulate further thinking about possible research and evaluation questions rather 
than as answers. 




The, gains in English usage areas may be related to some instructional and business 
characteristics . The clearest information about learner outcomes is on the four areas of English 
usage: reading, understanding, speaking and writing. The speaking and writing English 
outcomes were always more pronounced than reading and understanding. They were also 
pronounced when we examined their relationship to business and instructional characteristics. 
English usage gains were related to course length, with the mid-length courses (54-76 hours) 
showing the highest gains, the lengthiest courses (78-120 hours) showing the next highest gains, 
and the shortest courses (24-52 hours) showing the sm^lest gains. This suggests that short 
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courses are not advised for a predominantly ESOL population. The medium-sized businesses 
had higher gains than the sm^ler and larger businesses, with the latter two alternating on amount 
of gain across different outcome areas. 

Objective #4. Develop recommendations for project improvement 

Improve Consortium Function 

ACLS/Consortium staff should: 

• Support the CPET to become a more formally reflective governing body that is better prepared to 
promote institutionalization of programs in the Consortium. 

• Determine which programs are likely to continue, and assess what information and support other 
programs need to enhance the likelihood that they will also continue. 

• Support more evaluation activities in local PETs. 

• Consider organizing several state-wide sharings that focus on PET and CPET development. 

• Support the CWG to explore how student outcome data can be expressed in a uniformly 
quantitative way across programs. 

• Investigate whether the Consortium has leveraged more matching funds from businesses than 
other federally-funded programs. 

• Develop a dissemination plan for CWG materials; allocate staff time for dissemination; and use 
Consortium materials to promote program continuation. 



Improve Evaluation Activities 



Evaluators should: 



• Work with ACLS/Consortium staff to establish performance standards for Quality Indicators and 
develop new scoring process for Quality Indicators. 

• Use evaluation results as an agenda for discussion among ACLS/Consortium staff, CPET 
members and local staff and PETs to prioritize interest areas and develop hypotheses for testing 
with stronger data. 

• Encourage the CPET to discuss evaluation results in CPET meetings and consider how results 
might inform program improvement . 

• Gather more substantive data on how PETs are functioning than the Indicators PLUS protocol 
allowed in Year 2. 

• Support the CPET and local PETs to systematically assess both workplace and partnership 
outcomes. 



10 



Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page viii 



I. Introduction 

The Massachusetts Workplace Literacy Consortium . 

The Massachusetts Workplace Literacy Consortium (the Consortium) was funded for three 
years from November 1, 1994 to October 31, 1997 through the National Workplace Literacy 
Program (NWLP). The Consortium is composed of seven partnerships among business, labor 
and education that provided workplace education services at 26 businesses throughout the 
Commonwealth during its second year of operation.^ It is an association of small manufacturing 
companies, health care organizations, educational institutions and unions that intends to provide 
adult education services to approximately twelve hundred workers over the three year grant period 
for the purpose of upgrading work-related literacy skills. 



The partnerships are: 



Education Partner 


Business Partners 


Jewish Vocational 
Services 


C&K Components, Servolift/Eastern Corporation, Beth Israel 
Hospital and Children's Hospital, Fuller Mental Health 
Center, and Massachusetts General Hospital 


Quinsigamond Community 
College 


Jewish Healthcare Services, Beaumont at the Willows 
Nursing And Rehabilitation Center, and the Bolton Manor 
Nursing Home 


Worker Education 
Program/SEIU 


Metro West Medical Center, Jewish Memorial Hospital, 
Boston City Hospital/Boston University Medical Center, 
Jordan Hospital, St. John of God Hospital, and the Harvard 
Street Neighborhood Health Center 


Bristol Community 
College/ Attleboro Workplace 
Education Collaborative 


Robbins Company, Stern Leach, Jostens, Inc., Swank, 
Mason Box, and Helix Technology 


Labor Education Center at the 
University of Massachusetts, 
Dartmouth 


International Dryer Corporation and Lightolier Corporation 


Labor Management 
Workplace Mucation 
Program at the University of 
Massachusetts, Amherst 


University of Massachusetts, Amherst and Smith and Wesson 
Corporation 


Literacy Volunteers Network 


Holyoke Card and Paper Company and the Sealed Air 
Corporation 



Un January 1997 the number of participating programs is 25. 
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The Consortium is administered through the Adult and Community Learning Services 
Cluster (ACLS) of the Massachusetts Department of Education. Through ACLS, the Consortium 
receives the support necesstuy to develop, implement, evaluate and institutionalize its programs. It 
is the intention that the Consortium will continue to build a strong infrastructure of support for 
workplace education within the Commonwealth beyond NWLP funding. 

The governance structure of the Consortium is democratic. It is composed of the program 
Coordinators from each of the seven learning provider agencies which deliver services to the 25 
sites, ACLS staff, and representatives from the System for Adult Education Support (S ABES), the 
state's professional development agency for adult educators. This governance structure, called the 
Consortium Planning and Evaluation Team (CPET), meets regularly to review and address the 
issues attendant to providing education services in the workplace and to evaluate their own 
activities as a Consortium. The CPET receives guidance and advice from a twenty-four member 
advisory council, the Massachusetts Workplace Education Coordinating Council. The complex 
structure of the Consortium, its educational scope, and capacity to include large and small 
businesses from different sectors of the economy represent ten years of learning within the 
Massachusetts Department of Education about how to best provide workplace literacy services to 
employed workers. 

Review of the Consortium evaluation plan and its four objectives. 

The National Workplace Literacy Program requires that an independent external evaluation 
of all its projects be conducted. In November 1995 the external evaluators for the Consortium 
submitted an evaluation plan to the Secretjuy of Education, U.S. Department of Education, on 
behalf of the Massachusetts Department of Education, Massachusetts Workplace Education 
Program. The plan was subsequently approved and guided the evaluators in the second grant year. 
The evaluation plan is structured by four main objectives: 

1. Determine if the goals of the Consortium are being met. 

2. Determine the level of implementation of the Massachusetts Indicators of Quality for 
Workplace Literacy Programs and the relationship of the Indicators to learner outcomes, 
workplace outcomes and the program partnership. 

3. Determine the relationship between instructional methodologies and worker and 
workplace outcomes. 

4. Develop recommendations for project improvement. 



ERIC 
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This report summarizes the evaluation activities conducted in the second grant year and 
presents their results. Quantitative data are reported on Period 3 only, the six months from 
November 1, 1996 to April 30, 1997. It uses the four evaluation objectives as a framework 
through which the Consortium’s accomplishments and challenges can be assessed. 

II. Methodology 

The evaluation plan was designed around one central data collection effort, supplemented 
by several smaller efforts, that would simultaneously gather information on the extent to which 
Consortium goals were being met (Evaluation Objective #1) and on the extent of the presence of 

the Indicators of Quality for Workplace Education Programs (Evaluation Objective #2). 

• < , 

The Indicators PLUS protocol 

The central data collection effort was structured around a protocol called the "Indicators 
PLUS." "Indicators PLUS" derives its name from the "Indicators of Quality" which are quality 
standards established for six basic components of workplace education programs. The six 
components are: Partnership and Planning; Curriculum; Assessment, Evaluation and Outcomes; 
Support Services, Staff; and Administration. An example of an indicator within the component 
"Partnership and Planning" is: "There is a comprehensive plan for the program that is developed 
by all key stakeholders and reviewed regularly by them." (See Appendix A for a list of the 
Indicators and Appendix B for the Indicators PLUS protocol.) 

The central protocol is called "Indicators PLUS" because extra questions were added to the 
basic indicators protocol to cover all the areas needed for the goals evaluation. These extra 
questions focus on computer assisted instruction; distance learning; how the Consortium and 
Consortium Planning and Evaluation Team function as an association of programs; the economic 
and organizational context in which the business functions; productivity outcomes; and measures 
of ROI. Triangulation of data on partnership, curriculum and outcomes is attained by asking 
business partners, union representatives, teachers and workers to complete a shorter version of the 
Indicators PLUS protocol. 

The Quality Indicators were initially generated by adult educators, workers, and business 
and union representatives involved in previous funding cycles of the Massachusetts Workplace 
Literacy Program. They were then operationalized by the external evaluators in the first year of the 
Consortium. A significant amount of effort went into defining the Indicators. There has been a 
working assumption among programs in the Consortium and in prior NWLP-funded programs in 
Massachusetts that the Indicators describe the conditions that are necessary for a program to deliver 
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effective services. That assumption, however, has never been empirically tested. The Consortium 
evaluation gives us the opportunity to test this assumption by examining whether the presence or 
absence of the Indicators is correlated with desired outcomes at three levels: (1) workers’ 
educational gains; (2) productivity gains or improvements in quality of services arid (3) the quality 
of the business-union/education partnership. 

The Site Visit Process 

In the winter of 1996, the external evaluators administered the Indicators PLUS protocol in 
seven sites and provided support to Program Coordinators to collect data on their own in the 
remaining sites. As a rule, site visits were conducted over the course of one day. With one 
exception, representatives of all key stakeholder groups were interviewed in all the sites. These 
stakeholders include the Project Coordinator, business representatives, union representatives when 
a union is a partner, teachers, and a sampling of workers enrolled in the program. In a few cases it 
was not possible to interview everyone in person or on one day and the evaluators either made a 
second trip to a program site or conducted interviews by phone. The evaluators spent more time 
with the Project Coordinator — an average of two and a half hours — than with other stakeholders 
completing the Indicators PLUS protocol and the scoring sheet. Interviews with other program 
representatives lasted from approximately 30 to 90 minutes. 

The Scoring Process 

During the interview with the Program Coordinator, the evaluators asked the Program 
Coordinators to score each Indicator on a 6-point Lickert-type scale. The evaluators discussed the 
scores with the Coordinators and also scored the Indicators themselves without letting the 
Coordinator know that external score. During this process, the evaluators and the Coordinators 
familiarized themselves with the scoring process and the evaluators began to develop a sense of 
where their scores might differ from the Coordinators' scores. The Coordinators’ scores are for 
internal, formative use only. They provide a baseline against which Coordinators can monitor 
progress on each Indicator from Year 2 to Year 3. 

After the site visits were concluded and Coordinators had gained some familiarity with the 
protocols. Coordinators completed the Indicators PLUS and self-scores for all other sites. Other 
site and business/union personnel completed supplementary protocols. Once the additional 
protocols and self-scores were completed and mailed to the external evaluators, the external 
evaluators read them and scored them independently. The evaluators then met and compared 
scores for each Indicator on each protocol. When there was a discrepancy of one point or less, the 
rule was to assign an average of the evaluators’ ratings as the final score. When there was a 
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discrepancy of more than one point, the rule was to discuss the difference, defend the score and 
reach agreement on the appropriate score to assign. The result of this process was the set of "final 
ratings" used for all analyses of the Quality Indicators in this report, shown for each site 
(unnamed) in Appendix C and averaged across all sites in Table 16. The evaluators' initial ratings 
correlated at 0.83 using Spearman's rho^, indicating a high and statistically significant level of 
interrater reliability. More detailed descriptive information about the properties of the Quality 
Indicators are also included in Appendix C. 

We chose a 6-point Lickert-type scale because it: (1) enables statistical analysis with the 
outcome variables; (2) captures change well; and (3) is simple and familiar enough for local 
program staff to use. Through our pilots in Year 1 we learned that program staff were, in fact, 
able to discern the distinctions between the points. See Appendix D for the Self-Scoring Sheet. 

Assessment of strengths and weaknesses of the data collection and scoring processes 

The site visits provided us with a grounded sense of what was happening in seven sites, 
each representing one of the seven Consortium partnerships. This proved to be an effective way to 
prepare us to score protocols from other sites within the partnerships. In Year 3 ACLS/ 
Consortium staff is planning to facilitate a process to develop performance criteria for each 
Indicator. Performance criteria for each Indicator may simplify the scoring process; they will leave 
less to the interpretation of evaluators and make clearer to program staff exactly what criteria are 
associated with a particular score. 

Program Coordinators were given disks on which to complete their protocols. We 
anticipated that collecting data on disks would expedite the data gathering process but it became, 
instead, a technological burden in some cases. It is not easy to translate across different word 
processing programs and different versions of the same programs. A return to paper in Year 3 is 
likely the best solution, except for those few for whom the computer transfer worked well. 

Additional Data 

The Year 2 evaluation plan relied extensively on the use of quantitative data gathered 
through the National Workplace Literacy Information System (NWLIS). On account of funding 
recissions and other complications in the data base system, NWLIS data were not available to us. 
As a result, ACLS/Consortium staff developed a data base for the evaluators. See Appendix F for 
DOE data collection forms. This data base took what was usable from NWLIS and supplemented 



2 P<.000 on N=510. 
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NWLIS with new data when possible for Period 3 only, the period between November 1, 1995 to 
April 30, 1996. This composite data base for a six month period in the middle of the grant cycle 
offers ample descriptive data about program characteristics but littie data on outcomes. We 
performed our analyses on these data knowing that as many questions might be raised as answered 
by them, setting the stage for richer data collection and analysis in Year 3.^ This heuristic 
evaluation function is consistent with program evaluation theory elaborated by Michael Patton and 
others'*. Inconclusive results in Year 2 can inform the evaluation agenda for the subsequent year 
with questions that are based in analysis. 

We also used the following data sources to enrich our understanding of whether and how 
the Consortium goals are being met: 

• NWLIS summary data for Period 3 

• CPET minutes 

• Interviews with DOE staff 

• Documentation of the Curriculum Working Group 

• Distance learning documentation 

• Original grant proposal submitted by the Massachusetts Department of Education to the 
National Workplace Literacy Program 



Status of data collection and data availability. 

Although there were 26 sites in the Consortium for most of Period 3, data for two sites are 
incomplete. Jordan Hospital left the Consortium before the evaluators prepared their Quality 
Indicator scores! As a result, there are no Quality Indicator scores or analyses available for Jordan 
Hospital, although other descriptive data are available and are reported as appropriate. Conversely, 
Stem Leach has Quality Indicator and Quality Indicator PLUS data reported for it, but has no 
descriptive data in the DOE data base. 



In order to preserve confidentiality, programs are not identified by name in any analyses. 



3 For the first three reporting periods of the NWLP grants, all NWLP grantees were 
mandated to report to Mathematica Policy Research (MPR) using the NWLIS data system . 
NWLIS was developed by MPR specifically for the NWLP. NWLIS was designed to report data 
that was aggregated by partnership and not by individual site. Errors in the NWLIS system made 
some of the aggregated data unreliable. Although sites were not required to keep paper copies of 
NWLIS data input forms, most were able to refer to their records and provide sufficient data for 
this evaluation report. Missing data in Period 3 is, therefore, a reflection of the problematic 
NWLIS system and does not imply that Coordinators were remiss in keeping accurate records. 

“* Patton, Michael Quinn. Utilization-Focused Evaluation. Sage Publications, Beverly 
Hills, CA, 1977 . 

IG 
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III. Results 

The Consortium’s evaluation objectives provide a framework for reporting results. This 
section of the report is stmctured according to those objectives, with emphasis on the evaluation 
questions that emanate from each objective. 



Evaluation Objective #1:’ Detennine if the ^ 
BeihgMet. 










In this section of the report we review the five Consortium goals and assess the extent to 
which they are met. We list each goal and its indicators, identify, the. evaluation questions we seek 
to answer, present results, and offer conclusions about the extent of Indicator achievement. 

^Goal #1: Enhance the productivity and quality of services at 27 

businesses across the Commonwealth by improving the literacy 
skills of over 1,200 employees annually. 

Indicators: 

• a minimum of 1 ,200 employees are served annually 

• employees' literacy skills are improved 

• employers' productivity and/or quality of services are improved as a result of employees' 
improved literacy skills 

^Evaluation questions: 

• How many employees are being served? • Who are they? • What types of 

businesses/industries do employees work in? • What types of programs are they enrolled in? 

• What types of instruction are they receiving? 

This section serves as a baseline set of descriptive information about the businesses and 
industries in which the programs take place, characteristics of the programs, and the participating 
employees and their literacy gains for Period 3 of the grant cycle. Subsequent sections of this 
report will address the relationship between program characteristics and outcomes for the same 
period. Descriptive information provides the background that enhances the reader's understanding 
of the analyses of relationships between program characteristics and outcomes. This presentation 
is intended to provide an overall analytic description of the Consortium rather than a statement 
about individual sites or partnerships. 
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The Students 

During this period, 1,294 students^ were reported to have been served by 25 sites, for an 
average of just under 52 per site. The majority of sites (17) served between 5 and 50 students. An 
additional five sites served between 51 and 75 students. The two remaining sites served 163 and 
270 students. 

A summary of selected demographic characteristics of the students served is shown in 
Table 1. These results are based on the reports of 517 students (of the 1,294 students) from 23 
sites who reported any information to the DOE on a Learner Form. As the data in the table 
indicate, the average age of students is almost 42, though they range from 18 to 71 years old. A 
slight majority are female. The majority are not native to the United States, with over 80 percent 
born outside the U.S. Racially, the most numerous group is classified as White, with about one- 
fourth Other, presumably of Spanish speaking (since the same number that reported Other also 
reported themselves as being of Spanish descent in the next question on the form). Substantial 
numbers of Asians and African-Americans are also represented. English is reported to be spoken 
in about one-third (32 percent) of students' homes. The median level of schooling outside the 
U.S. is 10 years. For those bom in the U.S., the median years of schooling in the U.S. is 12 or 
more; for those bom outside the U.S., the median is 10 years in other countries and less than five 
in the U.S. The students have averaged over eight years in their present Jobs, however, 
suggesting general work stability of this group. 

In summary, the most significant features of these students with respect to their workplace 
educations may be that they are largely middle-aged adults, with less than a high school education, 
for whom English is not their primary language, and for whom U.S. culture is not their native one. 

Participating Businesses and Industries 

The types of businesses and industries represented in the Consortium, along with key 
characteristics, are summarized in Table 2. The slight majority are health care organizations, 
including hospitals, nursing homes, and mental health centers. Almost half are manufacturing 
companies. One is a university program for support staff such as maintenance, clerical, and food 
service workers. Of the 23 sites reporting the number of employees at their worksites, almost half 
are quite small with fewer than 250 employees, several have between 250 and 500 employees, and 



5 This is a sum of the numbers of students shown in each course reported on DOE’s course 
form. It is likely a duplicated count but is the best source available in this dataset for estimating the 
number of participants. .. ^ 

.1 O 
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Table 1. Demographic Characteristics of Employees Served 



Characteristic 


Number 

Reporting Percent* 


Mean S.D. Min Max 


Age 


457 


41.9 10.83 18 71 


Born in U.S. - Y 

-N 


89 19 

379 81 




Gender - M 

-F 


227 46 

267 54 




Race - White 

- African- Am 

- Asian 
- Native Am 

- Other 


201 43 

60 13 

88 19 

3 1 

112 24 




Spanish descent - Y 

-N 


112 24 

353 76 




English spoken at home -Y 

-N 


141 32 

303 68 




# years at present workplace 


381 


8.1 6.32 0 28 


# years' U.S. schooling** 

# years' other schooling** 


384 

404 


Median 

9 none 12 or more 

10 none 12 or more 


* These percents are of the total numbers responding to each particular question to the immediate left. The 
total pool of respondents is a maximum of 517 from whom Learner Forms were received. 

** Medians are reported for the schooling variables since the response categories on the data forms were in 
year spans up through 8 years» then in single years above 8. The figure for the # years' U.S. schooling is the 
median for those who have some schooling in the U.S.» removing those who had no U.S. schooling from the 
distribution. 



a substantial proportion (39 percent) have over 500. The median number of employees at these 
companies is 280. Furthermore, almost half the companies, 46 percent, are at least partly 
unionized, with slightly over half having no union membership. 

In summary, a significant feature of the participating businesses is that they represent 
almost equally the manufacturing and health service sectors and are small, medium and large in 
size. That almost half of the participating businesses are small indicates that the Consortium is 
serving the small business sector that it intended to serve without sacrificing delivery of . service to 
larger manufacturing or healt'n sector companies. 

.19 
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Table 2. Characteristics of Participating Businesses/Industries 



Characteristic 


Number 


Percent of 
Total 


Industry Type - Education (University support 






staff) 


1 


4 


- Health Care 


13 


52 


- Manufacturing 


11 


44 


Business Size - Small (<250) 


11 


48 


- Medium (250 - 500) 


3 


13 


- Large (>500) 


9 


39 


Percent of workers in union at site: - > 50% 


9 


38 


- < 50% 


2 


8 


- none 


13 


52 


Reasons for starting program: 






- reduce error and waste 


19 


76 


- change in work organization or processes 


19 


76 


- make greater use of ESOL employees 


17 


68 


- workers requested 


9 


36 


- labor agreement 


8 


32 


Incentives offered to workers: 






- partial paid release time 


17 


68 


- award ceremony on completion 


17 


68 


- award certificate on completion 


15 


60 


- complete paid release time 


7 


28 



Reasons for Starting Program 

We next considered the reasons the businesses had for instituting their workplace education 
programs and the incentives offered to workers to participate in them, using the data from Table 2. 
Overall, over three-fourths of the businesses started the programs to help reduce errors and waste 
and to help with changes in the work organization or processes. Most often the changes referred to 
include cross-functional training, working in teams, or participation in total quality management- 
type activities. Over two-thirds of the businesses also indicated that they want to make greater use 



O 



O 



Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page 10 



of their ESOL employees, consistent with the demographic characteristics of the students described 
above. 



One striking reason for instituting a workplace education program, though mentioned by 
only a little over a third of the sites, is that workers requested the training program. It is likely that 
this refers to training requested by workers through a union or required by a union contract. 
Examination of the nine companies for which this was stated as a reason for starting a program 
reveals that six were companies with greater than 50 percent unionization and two more were from 
those with less than 50 percent unionized; only one was from a non-union business. The question 
of how workers in unionized vs. non-unionized workplaces make their interest in education known 
to management might be worth pursuing. Unions provide a structure for workers to make their 
education needs known. Are workers in non-unionized workplaces given comparable 
opportunities to request education? What do those opportunities look like? If those opportunities 
do not exist, and the need for training is stated by management more often than by workers, does 
this dynamic affect workers’ participation in training? 

Companies' reasons for starting the programs vary somewhat by type of industry. The 
differences in each reason for the manufacturing and health industries are shown in Table 3. 
Reducing error and waste was more predominant among the manufacturing businesses, cited by all 
1 1 of those as a reason for the program, though only by about half the health businesses. The 
other three reasons were more frequently cited by the health businesses: labor agreement 
requirements, worker requests, and "other" for which the respondents wrote in some variation of 
job security. This last item, job security, seems noteworthy because of its predominance in the 
health businesses. Perhaps it is such a significantly higher concern there because of the uncertainty 
that has arisen in the health sector from recent and proposed federal, state, and private health care 
reforms, including the more publicized areas of Medicaid, Medicare, and managed care. 

Incentives 

The incentives generally offered to workers to participate in programs are fairly standard: 
partial paid release time from work for instructional time and an award ceremony and certificate 
upon completion are offered by about two-thirds of the businesses (68 percent). Seven companies 
(28 percent) offer complete paid release time from work. These are strong incentives for worker 
participation and they also bear upon the in-kind contributions which the companies make to their 
programs, which are described below as being quite high in many sites. The incentives offered to 
workers were compared across industry types and across business size, again resulting in no 

21 
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Table 3. Reasons for Starting the Program by Business/Industry Type 





Type of Industry* 


Reason for starting 


Health 


Manufacturing 


reduce error and waste 


7 of 13 


1 1 of 1 1 


change in work organization or processes 


8 of 13 


10 of 11 


attract new workers 


Oof 13 


1 of 11 


health and safety requirements 


0 of 13 


1 of 1 1 




6 of 7 


1 of 3 


labor agreement 






changes in workforce 


Oof 13 


1 of 11 


workers requested 


7 of 13 


1 of 11 


make greater use of ESOL employees 


7 of 13 


9 of 1 1 


other 


5 of 13 


Oof 11 


*The one university program was not included in 


this analysis. 


Since it was only 


one program in the Education category and it checked all the reasons in its response, 


it did not contribute to the comparison across either industry types or reasons for 


starting a program. 







patterns of differences. There is a relationship between partial paid release time and learning gains 
that is elaborated in Table 10. 

Instructional Characteristics 

General characteristics of instruction in the Consortium programs are shown in Table 4 
where the data indicates there were 138 courses offered during Period 3 across all sites, with an 
average of between five and six per site. The range was wide, however. One site offered only 
one; most sites offered from two to eight courses; and two offered significantly larger numbers, 16 
and 31 courses. The numbers of workers in each class averaged a little over nine, with a fairly 
narrow distribution around that figure. The courses also varied quite a bit in length, with an 
average of over 65 hours per course, and the distribution quite even within the extremes of 24 to 
180 hours.^ 

Additional characteristics of the course structure are that 1 16, or 88 percent, of the courses 
allow open entry, indicating the clear modal method for student access. Also, 57 percent are 
offered during the workday (not during lunch), with another 25 percent offered immediately before 



^ We suspect that the five courses that reported higher than 120 hours in length are 
reporting errors. Therefore, we have used the range of 120 in all tables to follow. 
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Table 4. Descriptive Statistics for Selected Course Characteristics 



Characteristic 


Mean/Site 


S.D. 


Min 


Max 


# courses offered (Total = 138) 


5.52 


6.12 


1 


31 


# students enrolled in each class 


9.51 


3.8 


2 


19 


# hours/course 


65.6 


28.6 


24 


180 



or after work (usually partly during the workday and partly before or after). Additionally, four 
percent (six courses) are offered at lunch, and 13 percent (18 courses) are offered at "Other" 
unspecified times. 

From NWLIS summary data for Period 3, we leam that 55 percent of students completed 
the courses in which they were enrolled. Seventeen percent dropped their courses and 25 percent 
have the status of “incomplete: needs skills/hours.” Twelve percent of those who did report a 
reason, report that family or health reasons (not including child care) prevented them from 
completing their courses. 

Other characteristics pertain more to the content of the instmction, and these are shown in 
Table 5. The primary types of content taught are ESOL, mostly at the intermediate level (45 
percent), followed by the beginner level (24 percent). A substantial amount, 16 percent, of pre- 
GED preparation is also taught. The mOst predominant emphasis within the courses is 
communications, in 38 percent of the courses. This is followed by reading and writing combined, 
then reading, writing and literacy each as separately reported emphases. The emphasis on 
communication is consistent with the ESOL course focus. 

doe's Course Data form asked respondents to indicate which instructional methods they 
used in each course from a list that included: small groups, teacher-led, computer-assisted 
learning, individualized, audio or visual aids, materials from the workplace, and other. Virtually 
all options were indicated for all courses, so results are not reported in the table. 

The vast majority of the courses, 89 percent, are reported to be developed uniquely for this 
program. Placement and assessment tools used most frequently are student interviews, portfolios, 
lEPs, and "other" methods. There is a distinctive lack of emphasis on use of tests, job- 
competency assessments, or supervisor ratings. Since the majority of the students are of limited 
English backgrounds and most of the instmction is in ESOL communications, student interviews 
make sense as a predominant and probably effective placement and assessment method. 
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Table 5. Frequencies of Instructional Characteristics (of 138 Courses) 




Characteristic 


Frequency 


% of Total 


Type of service 






- intermediate ESOL 


62 


45 


- beginner ESOL 


33 


24 


- pre-GED 


22 


16 


-GED 


8 


6 


- advanced ESOL 


7 


5 


-ABE 


5 


4 


- pre-literacy 


1 


1 


Primary emphasis of course 






- communications 


52 


38 


- reading and writing 


26 


19 


- writing 


16 


12 


- literacy 


12 


9 


- reading 


12 


9 


- other 


10 


7 


- reading and math 


5 


4 


- math 


4 


3 


- team building 


1 


1 


Most of curriculum uniquely 
designed for this course 


116 


89 


Placement Tools Used 






- student interviews 


113 


82 


- other 


63 


46 


- lEPs 


36 


26 


- portfolios 


17 


12 


- standardized tests 


11 


8 


- job-related competency tests 


10 


7 


- supervisor ratings 


6 


4 


Assessment tools used: - other 


83 


60 


- portfolios 


70 


51 


-lEPs 


45 


33 


- student interviews 


42 


30 


- Job-related competency tests 


20 


15 


- standardized tests 


9 


7 


- supervisor ratings 


6 


4 



2 4 
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Instructional Characteristics bv Industry Type 

Some instmctional characteristics differ by industry type, most strongly class sizes and 
primary course emphases. The manufacturing sites' courses are more often larger (56 percent are 
in the large, 10-19 students, category) than are the health sites' courses (46 percent). Courses 
with primary emphasis on reading, math, or reading and math are offered in substantially greater 
proportion at manufacturing than at health sites. Courses with primary emphasis on writing, 
communications or reading and writing are offered in substantially greater proportion at health than 
at manufacturing sites. 

Number of hours the course is offered was also found to vary somewhat by industry type, 
though not to the high degree of the prior two comparisons. The data for this relationship are 
shown in Table 6. Note that the education site has the highest average number of hours in their 
courses; the manufacturing sites have a slightly lower number of hours; and the health sites have 
the shortest courses. 

Table 6. Mean Course Length in Hours by Industry Type 





Education 


Industry T 
Health 


ype 

Manufacturing 


Mean 


66.2 


60.8 


64.3 


S.D. 


35.4 


19.5 


24.8 


N 


5 


99 


32 



The final comparison for instructional characteristic by industry type is that of type of 
service. The findings indicate a slight tendency for more GED services in manufacturing than in 
health sites and more Beginning ESOL services in health than in manufacturing. The results for all 
types of services are shown in Table 7. 

Administrative and Instructional Staff 

There are seven Project Directors who average 13 hours of work per week on the 
Consortium. There are six Assistant or Associate Project Directors who average slightly more - 
15 hours — of work per week. There were 53 instructors in the Consortium during Period 3. 

The 3 1 instructors for whom there is data on work hours worked an average of 13 hours per week. 
In addition, there were five instructors’ aides who averaged seven hours of work per week and 15 
volunteers who contributed a total of 560 hours, an average of 37.3 hours per volunteer. 
Volunteers functioned as “instructors’ aides” and “tutors.” o r~ 
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Table 7. Type of Service by Industry Type 





Industry Type* 


Type of Service 


Education 


Health 


Manufacturing 


Pre-literacy 






3% (1) 


ABE 




3% (3) 


6% (2) 


Pre-GED 


60% (3) 


15% (15) 


12% (4) 


GED 




3% (3) 


15% (5) 


Beginning ESOL 




28% (28) 


15% (5) 


Intermediate ESOL 


40% (2) 


47% (46) 


41% (14) 


Advanced ESOL 




4% (4) 


9% (3) 


* Cells show the percent of classes in each industry type with the numbers of 


students in the categories to the left. Numbers of courses represented are 


shown in parentheses. 









Of the 39 instructors for whom there is gender data, 33 (85 percent) are women. Thirty- 
eight of the 53 (72 percent) are white; only two are African-American; two are Asian. None is 
described as Hispanic. Eleven are described as “other.” Of the 31 instructors for whom there is 
data on educational attainment, 14 (45 percent) have a Master’s Degree but not a Ph.D.; and eight 
(26 percent) have some graduate credits. That means that 22 of the 31 (71 percent) have some 
graduate training. Seven have a four-year college degree only; one has some college, but not a 
degree, and one has a two-year college degree. The tasks routinely assigned to instructors 
included: teaching learning sessions; recruiting students; assessing students; collecting data for 
project evaluation; conducting Job task analysis; designing/adapting curricula; developing learner- 
developed educational plans; counseling employees on their educational development; working 
with worksite supervisors; promoting workplace education at the worksite; and “other.” 

These data confirm that Consortium instructional staff are predominantly white women 
with some graduate training who are called upon to perform many Job responsibilities that exceed 
any definition of traditional teaching. The data underscore the need for staff orientation and 
training, which both local and ACLS/Consortium staff provide. 
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■^Evaluation questions, Goal #1, continued 

• What are employees’ literacy gains? • What improvements in productivity and/or quality of 
services do employers realize as a result of employees' literacy gains? • What do programs 
cost? • Is ROI important to companies and unions? 

Learner Outcomes 

Learner outcomes were assessed for this evaluation, first, by having the students rate 
themselves at the start of a course and at the end of it in seven skill areas, shown in the left column 
of Table 8. We computed the difference between their start and end ratings as the gain score in 
each skill area. The validity of these ratings and gains will vary by course, for some skill areas 
match the course objectives better than others. Also, ratings were provided by from 158 to 181 
students on the various skill areas out of the total 1,294 students reported to have been served 
during Period 3. Thus, there are limitations to the generalizability of these data to all students 
served during the period. Given those limitations, however, the overall gain ratings are shown in 
Table 8. 

Table 8. Mean Rating "Gains" in Seven Skill Areas 



Skill Area 


Mean 

Gain* 


S.D. 


Minimum 


Maximum 


Number 

of 

Students 


Writing English 


0.16 


0.73 


-2 


3 


175 


Speaking English 


0.11 


0.67 


-2 


2 


176 


Solving problems/reasoning 


0.11 


0.64 


-2 


2 


158 


Understanding English 


0.1 


0.66 


-2 


2 


176 


Reading English 


0.09 


0.61 


-2 


2 


181 


Using math 


0.09 


0.75 


-3 


2 


162 


Working as part of a team 


0 


0.66 


-3 


2 


170 


Sum of gains** 


0.66 


2.95 


-12 


13 


181 


* Based on pre-post self-ratings on a 4-point scale: (1) poor, (2) fair, (3) good, (4) excellent. 

** The gains for each individual were summed over the seven skill areas. Those gains were then averaged 
for this and subsequent tables showing sums of gains. 



The results indicate some gains, particularly in writing English, but also to a fair degree in 
speaking, understanding and reading English, using math, and solving problems and using 
reasoning. The gains in English usage stand to reason since most of the courses addressed those 
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content areas, as described above. Moreover, working as part of a team was an area reported to be 
addressed by only one site, and thus the lack of gain there makes sense. The problem-solving and 
reasoning gains look as strong as the English gains, even though there are no programs that 
reported course focus in that area. Perhaps either the 10 courses reported as "other" addressed this 
content area or problem solving and reasoning gains are experienced as component parts of other 
gains. The positive sum of gains is shown at about 0.66. 

The sum of gains of 0.66 signifies that, overall, workers rated themselves higher at the end 
of Period 3 instruction on the seven skills areas than at the beginning. Workers were given a scale 
on which to rate their pre instruction status and their post instruction status. The scale for each of 
the seven areas is as follows: 1 = poor; 2 = fair; 3 = good; 4 = excellent. Thus the 0.66 gain 
represents a 22% gain of the scale’s range. It is a key finding of this evaluation that workers made 
noticeable learning gains in Period 3 alone. This means that the Consortium has made progress 
toward its goal of improving the literacy skills of workers. 

A second means of assessing learner outcomes was from a series of questions on the same 
learner assessment form that asked for reports on whether each of several other outcomes occurred 
for workers. These other outcomes and the frequencies with which they were reported to have 
occurred are shown in Table 9 for outcome areas in which 16 percent or more responded. As with 
the self-ratings in the skill areas above, these data are reported by a small proportion of the 
employees served during the period. Nonetheless, ninety-five percent of the respondents indicated 
that they learned what they wanted to, which speaks well for participant satisfaction with the 
courses. 

Table 9. Self-Reports of Other Outcomes 



Outcome 


Frequency 


N Responding 


% of Total 


Learned what wanted to 


197 


208 


95 


More responsibility added to job 


94 


207 


45 


Changed educational or career goals 


86 


210 


41 


Received a pay raise 


48 


203 


24 


Received award, bonus, or other recognition 


32 


200 


16 



Release Time and Learning Gains 

We examined whether the type of paid release time offered to workers to participate in the 
program made a difference in their learning gains. Specifically, we compared complete and partial 

28 
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paid release time sites. The results, shown in Table 10, indicate that the sites with partial paid 
release time had higher gains in the sum and in all outcome areas except problem solving than the 
gains in sites with complete paid release time. The sites with complete paid release time included 
five manufacturing, one education, and one health businesses. Those with partial paid release time 
included 10 health and five manufacturing. 

Table 10. Mean Learning Gains by Amount of Paid Release Time 





Paid Release Time 


Outcome Area 


Complete 

(N=105) 


Partial 

(N=209) 


Reading English 


0.05 


0.14 


Understanding English 


0.03 


0.18 


Speaking English 


0.05 


0.25 


Writing English 


0.14 


0.28 


Teamwork 


-0.07 


0.09 


Math 


0.02 


0.13 


Problem solving 


0.01 


-0.04 


Sum of gains 


.023 


1.03 



Workplace Outcomes 

The primary source of workplace outcome information is from site personnel's responses 
to open-ended questions on the Indicators PLUS Protocol, and the teacher, business partner, 
and/or union protocols, either through interviews in the sites we visited or through written 
responses. From those written or oral responses, the evaluators content analyzed and summarized 
the major categories of workplace outcomes, then tallied their occurrence at each site. The 
categories and their frequencies of occurrence are shown in Table 1 1. The types of information 
included in each category are; 

•Improved communication, including more reading of work orders, less need for 
interpreting and workers ask more questions about the status of the department and of sales 

• Improved involvement in teams and work processes, including making suggestions 
and explaining to others what's needed to do a job 

• Less scrap and rework, fewer problems with quality; fewer errors due to 
misunderstanding; and more material control accuracy; improved quality of service 

• Improved self-confidence 

• Improved safety, in one case resulting in lowered insurance premiums for the company 
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• Improved participation, including less absenteeism & turnover; higher rates on — and 
more participation in — performance reviews, and reports of praise from supervisors 

• More involvement in the union, including writing grievances 

• Program created an environment that is supportive of training and 
institutionalizing the program 

• Workers are promoted 

• No visible results 

Table 11. Frequencies of Workplace Outcomes 



Outcome Category 


Number of 
Sites 

Reporting 


Percent 

of 

Sites* 


Improved communications 


15 


75 


Improved participation 


7 


35 


Less scrap & rework 


6 


30 


Improved self-confidence 


4 


20 


Improved safety 


4 


20 


More union involvement 


3 


27** 


Improved teamwork/work process involvement 


2 


10 


^ Out of the 20 who reported any outcome information. 

Out of the 1 1 who reported that at least some employees were union members 



Note from Table 1 1 that improved communications was the most predominant outcome in 
the workplace, consistent with the primary course emphases and types of services most often 
offered. Reports of improved communication may have been even more numerous had we asked 
respondents if improved communications was a visible workplace outcome. As noted, we asked 
open-ended outcomes questions. When answering an open-ended question about workplace 
outcomes, respondents might emphasize a point that is salient at the time and overlook something 
that would be noted if prompted. 

Reports of improved participation in teams and work processes and reduced scrap and 
waste were the two next most frequently reported outcomes. These outcomes are consistent with 
the two most frequently given reasons for starting the program which were reducing error and 
waste and changes in the work organization or process. 

Looking at the workplace outcomes from the perspective of sites, the 20 that reported any 
outcomes reported them in anywhere from one to four of the categories. More specifically, five 
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sites reported outcomes in only one category, nine sites in two categories, and three sites each 
reported them in three and four categories. This means that the majority (17 of the 20) report 
multiple workplace outcomes. 

Cost bv Site 

Actual cost data were not available by site. Thus, we used the budgeted figures for each 
partnership and pro-rated the budget for one-half of Year 2 among the sites within each partnership 
according to the numbers of students served to derive the estimated budget per site. We then used 
that estimated amount for each site to calculate the cost per student and cost per student contact 
hour based on the actual reported numbers of students served and numbers of student contact 
hours. 



The overall cost data for all sites are shown in Table 12. The mean cost per site is $35,939 
of which $14,908 is from the NWLP grant and the remainder is from company cash contributions 
(largely, if not exclusively, paid release time for instruction) and in-kind contributions of the 
education or business partners or both. The vast majority of those non-public funds are from 
company cash contributions. The programs cost about $875 per student during Period 3, 
including $348 of public funds and $527 of non-public funds. The cost per student contact hour is 
about $14.23, with less than half that from the NWLP funds and over half from the company cash 
and other in-kind contributions. 

When we compare the costs by business/industry type, shown in Table 13, the overall 
result is that the health sites have considerably larger budgets than the manufacturing projects, but 
also have lower costs per student and per student hour for their non-public and their total funds. 

The manufacturing sites use considerably less NWLP funds and a slightly lower proportion 
of cash contributions per student and per student contact hour than do the health sites. This result 
may be reflective of the higher wages earned by the students in the manufacturing sites compared 
to the health sites, thus making the manufacturing sites' relative cash contributions much higher, all 
other things being assumed equal.^ 



From the data reported by individuals, the average hourly earnings of 176 health care 
workers is $6.98 while that of 102 manufacturing workers is $7.33. This tends to support the 
hypothesis. The data were reported for less than half the participants who reported individual data 
and about 21% of the participants' served, however, so they do not seem representative enough to 
warrant any conclusive statement about the hypothesis. 
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Table 12. Estimated Costs Across All Sites (in dollars) 





NWLP 

Funds 


In-kind & 
cash match 


Total 


PER SITE 
Mean 


14,908 


21,031 


35,939 


minimum 


1,882 


1,003 


2,885 


maximum 


41,498 


111,497 


144,424 


PER STUDENT 
Mean 


348 


527 


875 


minimum 


133 


116 


249 


maximum 


630 


1,807 


2,340 


PER STUDENT 
CONTACT HOUR 
Mean 


5.81 


8.42 


14.23 


minimum 


2.97 


2.74 


5.89 


maximum 


14.80 


23.63 


30.61 



Table 13. Mean Estimated Costs Per Site for Business/Industry Types 





Business/Industry Type 




Health 


Manufacturing 


NWLP funds 

budget 
cost/student 
cost/student contact hour 


19,986 

392 

5.85 


7,268 

284 

5.65 


Non-public funds 

budget 
cost/student 
cost/student contact hour 


18,703 

415 

6.36 


12,778 

432 

8.10 


Total funds 

budget 
cost/student 
cost/student contact hour 


38,689 

808 

12.21 


22,827 

841 

15.12 
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We also explored the cost data across small, medium and large businesses, similar to the 
analysis in Table 13. The trend was that the monies budgeted, both from public and non-public 
funds, were greater for the larger companies and that the costs per student and per student contact 
hour were lower for the larger companies. This result, however, may be largely a function of the 
way in which the partnership funds were allocated to sites, as described earlier. That is, the larger 
companies generally had more students and thus were "assigned" a larger prorated share of the 
partnership budget. Since they had more students, dividing their budgets by their larger numbers 
of students and contact hours gave them lower costs per student and per contact hour. Since the 
logic for the analysis is essentially circular, we did not present the table of results or discuss it any 
further. 



After attempting to analyze cost data and derive some conclusions about them, it is clear 
that the data herein are inadequate. Insofar as cost considerations are important to state or 
partnership policy decisions, better cost data are needed if there is to be any possibility of 
evaluation or analysis informing those decisions. As a result we recommend that the DOE set up 
and implement a system for reporting budgeted and actual cost data by site. 

Is ROI Important? 

We investigated whether formal Return on Investment (ROI) measures are in place in any 
Consortium programs and are considered of value. There is a working assumption that business 
partners are interested in measurable outcomes, and in ROI measures in particular. Good ROI 
measures enable Human Resource Managers and other trainers and educators to use evidence and 
not just intuition to argue that worker education benefits the workplace. Demonstrating ROI has 
been considered a key element of program continuation. If real financial benefit to the company 
can be demonstrated, then the company might be more likely to support its own program after 
federal funding is finished. This working assumption does not hold true among Consortium 
businesses to the extent expected. 

As noted in Table 14, nine out of 22 business partner respondents expressed an interest in 
ROI. Six business partners reported that ROI is not important. Seven business partners, those 
shown in Table 18 with missing data, failed to answer questions about ROI, even when they 
completed other parts of the protocol, possibly signifying lack of interest in ROI. This makes a 
total of 13 business partners who either answered directly that ROI does not interest them or about 
whom we might infer the same, representing more than half of the programs. 
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Table 14. ROI by Type of Industry 





Type of Industry 


Opinion of ROI 


Education 


Health Care 


Manufacturing 


Want ROI 


1 (4.5)* 


4(18) 


4(18) 


Don’t want ROI 




3(14) 


3(14) 


Missing Data 




4(18) 


3(14) 


* Twenty-two respondents completed protocols in which questions on ROI appeared. Cells show 
the numbers and, in parentheses, percent of response of the total 22. 



The explanations that business partners gave for not having an interest in ROI range from 
vehement to casual. Vehement explanations include: "Fabricating ROI formulas is a wasted 
effort.... Fortunately, our management sees the benefits of operating a [workplace education] 
program and does not require hard data," and "Autocratic management systems need piles of data. 
But businesses don't support programs like this if they didn't expect ROI. We ‘know’ that this 
program is a success. We don't want to take the time to collect data on what we know is 
working." 

Reasons given for why ROI is important include: "[ROI] is very important because it's 
hard to convince upper management of the benefits without hard data." "We want some data to 
support the program after federal funding." "We would like to have fourth level Kirkpatrick 
information in place but this is not available for other programs either." [ROI] is important but 
we've explained it in intangibles. We need hard data." One respondent said that his program has 
not been able to calculate ROI because of "lack of objective measurements." He suggested the 
following data collection method: "At the end of each [instructional] period, there should be short 
written statement of the employee's commitment, areas of improvement, and areas needing 
improvement — a report card." 

There is only one company that has in place what we might call an ROI formula. This 
company collects scrap and rework data every day and then calculates it weekly and monthly. 
Scrap and rework continue to be reduced, and the value of the reduction is attributed to worker 
participation in an education program. Only one other site has real plans to have an ROI measure 
in place in the future. Less scrap and rework; fewer problems with quality; fewer errors due to 
misunderstanding; and more material control accuracy — these are potentially the most easily 
measured workplace outcomes that can be turned into an ROI measure. While some companies 
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and organizations may not have systems that track scrap, rework or errors with ease, insofar as 
they interested in ROI, it seems worth the effort for Program Coordinators to assess what data 
companies are already collecting and determine if those data might be used to document workplace 
outcomes for the program and to calculate ROI. 

It is likely that some employers will continue to value rigorous ROI measures and others 
will not. In workplace education programs in other states, we have noted a striking correlation 
between disinterest in ROI and implementation of quality management. The more participatory the 
management style, the less interested managers are in traditional ROI. These managers trust that 
more education of the right kind - that workers help to define - will result in workers developing 
the skills they need to do their jobs better. These “generic workplace benefits” that result from 
improving workers’ education levels are supported by findings from the National Employer 
Survey. "Recent research using household surveys of workers suggests that there is an eight 
percent return to workers (in the form of higher wages) for each additional year of schooling. The 
EQW-NES documents that increases in the average education of an establishment's workforce has 
the same payoff for employers in the form of increased productivity."® 

In Year 3, it may be useful to characterize Consortium businesses as high productivity or 
traditional and analyze the relationship between this characterization and business leaders’ opinion 
about ROI. We might discover that traditional ROI measures are not what's needed to persuade 
management in the high productivity workplace that workplace education programs are worth the 
investment in them. 

■^Conclusions: Goal #1. 

• The Consortium is serving the population it intended to serve. The Consortium is serving a 
primarily intermediate ESOL population, mostly white middle-aged adults, a slight majority 
being female, with over 80% bom outside the US, who have less than a high school but 
more than an elementary education and who have enjoyed stable employment for an average 
of eight years. These are likely the people who will carry the burden of implementing 
organizational change at the floor level in both the manufacturing and service industries, as 
they are skilled enough and experienced enough to do so. The Consortium has met its goal 
to serve 1,200 workers per year. It has served 1,294 (duplicated count) in Period 3 alone. 



® “The Other Shoe: Education's Contribution to the Productivity of Establishments, A 
Second Round of Findings from the EQW National Employer Survey. Philadelphia, PA: National 
Center on the Educational Quality of the Workforce, undated, page 2. 
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• Workers have achieved noticeable learning pains A key finding of this evaluation is that 
workers achieved noticeable learning gains in Period 3. Analysis of learner self ratings in 
seven skill areas at the start and end of a course indicates that workers have achieved gains 
particularly in writing English, but also to a fair degree in speaking, understanding and 
reading English, using math, and solving problems and using reasoning. A sum of gains of 
0.66 signifies that, overall, workers rated themselves higher at the end of Period 3 instruction 
on the seven skills areas than at the beginning. A 0.66 gain represents a 22% gain of the 
scale’s range. A series of questions also asked workers for reports on whether other 
outcomes occurred. As with the self-ratings in the skill areas referenced above, these data are 
reported by a small proportion of the workers served during the period. Nonetheless, ninety- 
five percent of the respondents indicated that they learned what they wanted to, which speaks 
well for workers’ satisfaction with instruction and suggests the achievement of learning gains 
as well. Thus we conclude that the Consortium has made progress toward its goal of 
improving the literacy skills of workers. 

• Some positive impact on the workplace has been achieved. Business partners and site 
personnel from 17 of 20 sites reporting state that there has been some positive impact of 
worker participation in a program on the workplace. They state that the main areas of 
improvement are employee communications, participation in team work and work processes, 
and reduced scrap and rework. Fifteen of the 20 respondents (75 percent) cited improved 
communication as a workplace outcome, consistent with the Consortium’s ESOL 
demography, its primary course emphases and the types of services most often offered. 

Seven of the 20 respondents (35 percent) reported improved participation in team work and 
work processes, and six (30 percent) reported reduced scrap and rework. It is likely that 
improvements in communication and participation translate into improvement in productivity 
and/or quality of services. Reduction in scrap and rework - even in small quantity - usually 
translates directly into improved productivity. We conclude that the Consortium has made 
some progress toward its goal of improving productivity and/or quality of services for 
participating businesses. 

• The Consortium is making wise use of federal dollars. The overall mean cost of a program 
per site is $35,939, of which $14,908 is from the NWLP grant and the remainder is from 
company cash contributions and in-kind contributions of the education or business partners 
or both. The programs cost about $875 per student during Period 3, including $348 of 
public funds and $527 of non-public funds. The cost per student contact hour is about 
$14.23, with less than half that from the NWLP funds and over half from the company cash 
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and other in-kind contributions. We conclude from this cost analysis that the Consortium is 
using National Workplace Literacy Program monies in the way that they were intended — to 
seed workplace education programs that have the potential for continuation beyond federal 
funding and to leverage monies and services from companies that support these important 
programs. We would like to see figures that compare use of NWLP funds and matches in 
other NWLP-funded programs. Perhaps Massachusetts sets a standard for leveraging 
corporate dollars and for stretching federal ones. 



■^Goal #2: Build the capacities of individual partnerships to 

implement responsive workplace literacy programs using a 
Consortium support model. 

Indicators: 

• Local PETs are established 

• Local PETs receive training and technical assistance in order to function according to 
Consortium standards 

• Consortium meets regularly, identifies key issues which PETs are dealing with, and develops 
— and makes available — appropriate interventions/ supports 

• Consortium evaluates its activities. 

• Program staff participate in distance learning opportunities (mini-course and Picture-Tel) and 
benefits from such participation 

■^Evaluation questions: 

• Have all programs established a PET? • Who comprises the membership of the local PETs? 

• How often do PETs meet? • What work are PETs undertaking? • What are the PETs key 
accomplishments/challenges? • Are PET members receiving the training and technical 
assistance they need to meet their challenges? • Is the Consortium meeting regularly, 
identifying key issues which PETs are dealing with, and developing — and making available — 
appropriate interventions/supports? • Has the Consortium established a process for evaluating 
its own work? What is that process and how is it proceeding? 

The Consortium Support Model 

In the Massachusetts Workplace Literacy Consortium, the term “Consortium” refers to the 
collection of partnerships and programs that deliver workplace education services. The 
Consortium support model is one in which representatives of the seven participating partnerships 
meet regularly with ACLS/Consortium staff to oversee the operation of the Consortium and 
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determine how to provide support to local sites. Hypothetically, the governance structure of the 
Consortium mirrors the governance structure of programs at the site level; and hypothetically, 

PETs and the CPET are democratic environments in which multiple perspectives about program 
purpose and implementation can be voiced, and resolution — in the form of a functioning program - 
- can be achieved. At both the Consortium and site levels, representatives of stakeholder groups 
are charged to plan and evaluate their programs. 

At the site level. Planning and Evaluation Teams (PETs) are usually composed of business 
and union representatives, workers, teachers, and the site Coordinator, while the Consortium 
Planning and Evaluation Team (CPET) is composed of project Coordinators, ACLS/Consortium 
staff and the System for Adult Education Support (SABES) representatives. The practice of 
governance through local PETs varies from site to site depending on factors such as extent of 
participation of workers and supervisors, management style of the company, union involvement, 
degree of commitment to education, etc. Despite these differences, some consistency of 
governance process is achieved within each of the seven partnerships through participation of 
partnership Coordinators in the local PETs. For example, the coordinator of five programs 
supported through the SEIU facilitates PET meetings at each site and insures that a similar 
governance process is followed. In the text that follows, we first address how local PETs are 
functioning. Then we address how the CPET is functioning. 

Unless otherwise stated, data for these sections of the report were taken from; 

• The Indicators PLUS protocols completed by Site Coordinators, and other related 
protocols completed by teachers and business and union representatives 

• Interviews with ACLS/Consortium staff 

• CPET minutes 

• The text of the original proposal submitted by the Massachusetts Department of Education 
to the National Workplace Literacy Program. 

How Local PETs are Fimctionine 

All sites have established a PET. Twenty-one of 25 PETs (84 percent) are composed of 
representatives from all stakeholder groups — workers, teachers, union when a union is a partner, 
and company management. Most of the PETs include supervisors. PETs tended to meet once a 
month in Year 1 but frequency of meetings decreased in Year 2. PETs appear to be functioning as 
intended - as a hub where representative stakeholders plan, implement and evaluate their 
program's activities. 
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Nonetheless, PETs experience challenges, especially with harnessing full stakeholder 
representation. Challenges include: 

• Four PETs do not have full stakeholder representation. Of those four, three are missing 

company managers. 

• In one case, the manager left and was not replaced. Management's preoccupation with 
intensive restructuring of this workplace cast doubt on the viability of the partnership, 
although the program continues without management representation. 

• In the second case, the manager was laid off and not replaced. This created a sense of 
deep uncertainty about the program's future and affected the extent to which 
supervisors saw themselves as partners in the program. Responsibility for the program 
was shifted to the participating union, and the program continues under union auspices. 

• In the third case, the manager was on maternity leave until recently and, during her 
absence, the team did not meet regularly. The situation at this site was further 
complicated by not including supervisors in the goal setting process at the beginning of 
the program. Their initial lack of buy-in affected the extent to which PET members saw 
themselves as "partners." 



• Of the four PETs that do not have full representation, one reported lack of regular worker 
representation. This site recruited two workers, one of whom soon left the job. The 
remaining worker representative became ill. The worker who left was eventually 
replaced, but this person also became ill and did not come to meetings regularly. 



•Two PETs reported problems with supervisor support but offered no explanation. 



Reports of problems with harnessing full stakeholder representation are so few that it could 
be argued they point to the success of the PET structure rather than to its shortcomings. It is, 
however, clear that the process does not work equally well in all programs. Although only one site 
reported a problem with worker representation, we understand from anecdotal reports from 
Coordinators and teaching staff that other programs have integrated workers into PET meetings 
with some difficulty, especially workers whose native language is not English. Similarly, we 
understand that full supervisor support has not been forthcoming in all programs. 



ACLS/Consortium staff suggest that, while PETs generally serve programs well by 
offering a model of collaborative work in the workplace, they could work better. For example, 
they could become more of a venue for expanding the domain of workplace education from skills 
acquisition to application of skills — the place where linkages between what happens in class and 
what happens in the work process are made. ACLS/Consortium staff also suggest that teacher and 
worker participation in PETs could be improved. Staff indicate that teachers are sometimes 
conflicted about how to participate fully in a PET. If teachers feel that they are advocates for 
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students, and workers do not feel they can freely communicate their opinions about program 
purpose or operations without reprisal from management, then teachers might prefer to remain 
silent rather than voice a similar opinion. For example, workers - and teachers by association - 
might be reluctant to discuss how a “communication problem” among workers is really a “listening 
problem” among supervisors. It can take time and skill to coax these issues put into a group in a 
form that it can manage, even in organizations that employ quality management processes. 

The data do not indicate how widespread the reluctance of workers and teachers to speak 
up in a PET about potentially risky issues is, but reports of this dynamic are not new. It has been 
noted at other times in the six years since the Massachusetts Workplace Education Initiative 
required its programs to establish PETs as governing bodies. Although it is not formally 
documented in the Consortium’s second year beyond Consortium staffs oral reports, this brief 
focus on the problem allows us to suggest that PETs might benefit from further training. Through 
the Consortium, old problems might translate into an opportunity to explore and advance PET 
functions. Now that PETs have established themselves in a fundamental way as a program hub, it 
might serve the Consortium to address current PET issues in a statewide meeting. Training that 
focused on establishing a PET and on planning and evaluation activities was provided by 
Coordinators in Years 1 and 2. In Year 3, in addition to addressing the dynamics of teacher and 
worker participation in PETs, training might also address the extent of worker representation in 
PETs, dealing with conflicting agendas among stakeholders, modeling participatory management 
process in a PET, and moving PETs forward on a post-federal funding trajectory. 

The Consortium PET as Governing Body 

The CPET is a ten-member group composed of the Consortium’s Coordinator and 
Assistant Coordinator, Coordinators from the seven participating partnerships, and a representative 
of the System for Adult Basic Education Support (SABES). In order to assess how the CPET is 
functioning, it is necessary to review the services and supports that the Consortium stmcture was 
intended to provide to individual programs and the products it was intended to produce. 

Theoretically, the Consortium stmcture strengthens the ability of each partnership and each 
site to address and meet their needs. It enables a degree of effectiveness and impact of services that 
is greater than if the partnerships tackled issues independently. This is especially the case with 
small businesses whose more limited resources typically restrict them from sponsoring programs 
with the same level of benefits and services enjoyed by larger companies. Specific services and 
supports that the Consortium offers include; training, technical expertise, capacity building, and 
institutional support. The products that the Consortium is producing are numerous. They include 
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workplace education curricula that are both industry-specific and generic to the workplace; staff 
development training videos for local and national distribution; computer-assisted learning 
programs for the health care industry; and documentation of the uses of video-conferencing as an 
instructional aide. These products become resources to participating programs. The process of 
creating them itself builds capacity for service delivery. 

Ideally, the CPET governs the Consortium. Our data collection to date enables us to 
review the CPET governance process to some extent. However, we are not able to answer fully as 
yet the question that underlies the Consortium’s and CPET’s purpose: has the collaboration of 
service providers and companies in a Consortium stmcture, and the presence of the CPET itself, 
enabled programs to deliver a scope and quality of services that they could not have delivered 
individually? 

Review of CPET minutes for the twenty month period between March 1995 and November 
1996 confirms that the CPET meets for four hours once a month. While local issues are addressed 
in program updates. Consortium-related management issues dominate the agenda, especially; 

• The function of the CPET 

• The range and limits of its authority to govern 

• How funds will be distributed and spent 

• Use of distance learning opportunities 

• Evaluation and data collection at the site and CPET levels 

• Institutionalization 

• Reporting requirements 

These issues are central to the Consortium’s mission. They are complex and demanding 
and parallel the main program items in the Consortium’s Plan of Operation. (See Appendix G for 
the Consortium s Plan of Operation.) Coordinators’ protocols and staff interviews suggest that the 
CPET’s content agenda was so complex and demanding that it absorbed the CPET’s attention, 
leaving little time to reflect systematically on its creative potential as a governing body. 

When it first convened, the CPET had only the mandate from the original grant proposal to 
guide its way - that it provide for the Consortium as a whole what PETs provide for local 
programs: a stakeholder-based process for planning and evaluating all aspects of program 
delivery. PETs, however, received training and support from Coordinators to implement a formal 
planning and evaluation process. The CPET did not receive comparable training for its own 
planning and evaluation process. Faced with myriad reporting requirements and having to comply 
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with the many facets of the Plan of Operation, it is likely that receiving training for its own 
planning and evaluation process (separate from training provided to support Coordinators work 
with local PETs) seemed more burdensome than liberating. Although the CPET has evaluated its 
meetings and some of its programs, it has not given itself the opportunity to step temporarily 
outside of the grant framework and employ a formal planning and evaluation process to reflect 
anew on its goals and possible activities. This has resulted in less proactive programming and 
decision making than might have been expected at the start of the grant cycle. 

When ACLS/Consortium staff reflected on what they would have done differently in the 
first eighteen months of the grant period, there was agreement on the following: 

(1) Establish a process that allows the CPET to gain an identity of its own, into which new 
members could be welcomed 

(2) Clarify questions about the extent of the authority and autonomy of the CPET. Is it 
really a governing body with decision-making powers? What is its relationship to the 
ACLS? 

(3) Develop a workplan that would have translated the Plan of Operation into a “digestible” 
plan 

(4) Hold longer CPET meetings, especially in the very beginning 

A well-orchestrated planning and evaluation process in the CPET would have helped CPET 
members to accomplish the items listed above. Perhaps the time is ripe to introduce a planning and 
evaluation process to guide CPET members’ decisions about what they want to accomplish 
between now and the end of the grant period, and beyond. The CPET has agreement that its three 
goals are: (1) compliance; (2) capacity building; and (3 institutionalization. Having reached this 
agreement, perhaps plans can be made to insure that the goals are met. 

Evaluation questions, Goal #2, continued 

• Which program staff have participated in distance learning opportunities and how have they 
benefited from such participation? 

Distance Learning: Opportunity and Challenge 

In the Consortium, “distance learning” refers to any one of three telecommunications or 
computer technologies through which workplace education-related instruction may be delivered 
potentially to large audiences at a distance from the point of origin. Distance learning is one of the 
ways that the Consortium is disseminating its learnings and can thus be understood as part of the 
achievement of the Consortium’s’ Goal #5: Produce and disseminate information on exemplary 
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workplace partnerships, delivery models, curricula, and industry-specific literacy materials. 
However, we discuss distance learning in Goal #2 because it is also a way in which the 
Consortium has built the capacity of individual partnerships to implement responsive workplace 
literacy programs. The three “distance learning” technologies are: (1) televised mini-courses for 
adult educators, aired nationally; (2) Lexicon, customized computer-assisted learning progrruns; 
and (3) Picture-Tel, two-way video conferencing. 

Televised mini-courses 

The mini-courses were offered in two consecutive autumns: September-October 1995 and 
September-October 1996. In the autumn of 1995, ACLS/Consortium staff produced seven hour- 
long video segments that were advertised nationally and delivered via satellite. The segments were: 
Laying the Groundwork; Program Design; Planning and Evaluation; Approaches to Workplace 
Curriculum Development; Curriculum in Action; Creative Materials Development; and Future 
Directions. In the autumn of 1996, ACLS/Consortium staff produced three hour-and-a-quarter 
video segments that were advertised nationally and delivered via satellite. The segments were: 
Assessment and Evaluation; Innovative Teaching Practices; Prioritizing Worker Involvement. The 
mini-course (the singular verb “mini-course” refers to the collection of courses offered in any year) 
provides a summary of good workplace education practices developed over the ten years that the 
Massachusetts Workplace Education Program has been in operation. It was developed by the 
Consortium’s Assistant Director to meet the needs of workplace educators in Massachusetts in a 
traditional live-teaching format. Response to the course was very favorable and it seemed an easy 
step to make the course available nationally through video. 

Experience in adult education, however, does not necessarily translate into success with the 
video medium. The 1995 segments are competent and convey much useful information. They 
were nonetheless produced under duress, with limited production support from the production 
company hired to assist, and the strain that students suffered is often visible. A “best of’ tape — 
one tape that collects the best of six hours into one hour of broadcast — will be compiled and will 
likely be of great value as a training tool for workplace educators. 

ACLS/Consortium staff lacked experience with TV. Coupled with limited professional 
production assistance, and the pressure to create a product when the Consortium was barely a year 
old, this made the experience difficult. In 1996, another three segments were produced that 
benefited from the prior year’s learnings. ACLS/Consortium staff had a clearer idea of what they 
wanted to convey (disseminate work of the Consortium), who the audience was, and how to make 
fuller use of the medium, including call-ins, graphics and images. The segments were increased in 

43 



Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page 33 



length to one-and-a-quarter hours, making it more worthwhile for viewers in the western states to 
drive great distances to a down-link site. While there are frustrations built into the distance 
learning format — you cannot know how many people you are reaching or what their reactions are 
unless you set up a system for gathering this information — written evaluations suggest that the 
experiences of Massachusetts workplace educators were well-received by workplace educators in 
other parts of the U.S. 

Lexicon : Computer-assisted Instruction 

The Consortium hired a computer consultant to create six units of computer-assisted 
instruction with a focus on basic literacy skills in the health care industry. The consultant 
completed five instructional units for the whole Consortium. The units are: (1) Introduction to 
Workplace Education Programs; (2) Back Safety; (3) HIV/AIDS; (4) Performance Evaluation; (5) 
Team Work. 

Considerable effort was put into the Lexicon project during Years 1 and 2. One of the 
accomplishments was learning about the kind of computer-assisted instruction that works for 
Consortium teachers and students. Coordinators’ protocols and interviews with 
ACLS/Consortium staff suggest that the Lexicon project was challenging in several ways and 
needs to be adapted to the following concerns: 

• First, each partnership was granted only one MAC computer on which to deliver 
Lexicon’s computer-assisted instruction. Demonstrating the utility of computers in the classroom 
with limited access to computers is impractical. 

• Second, secured facilities were not available at all the sites and teachers or Coordinators 
had to transport computer equipment. 

• Third, the software was designed at an English literacy level that was too high for most of 
the students. 

These difficulties may have been worked out had Coordinators and teachers been formally 
supported to gather together and discuss how the Lexicon system might best serve their students. 
As described below, in Year 2 teachers were paid to gather and discuss how they might use 
Picture-Tel in ways other than those the original grant suggested. Teachers then took ownership of 
the project and became invested in the creative use of the technology. A group of teachers may 
have continued discussing creative approaches to Lexicon suggesting that all computers be installed 
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in one program where security is not an issue, and that intensive computer-assisted learning 
opportunities be provided and tested in that program. Even such a strategy would have had to be 
well-timed. ACLS/Consortium staff stressed that in Year 1, program staff did not perceive a need 
for exploring computer-assisted instruction. They were more concerned with basic issues like 
“whether students were coming to class or not.” 

Lexicon was not presented, however, as an opportunity for teachers’ to explore the uses of 
computer-assisted technology. Rather, it was presented as an essentially non-negotiable 
component of the Consortium, something that had enjoyed success in other programs and that 
would benefit the Consortium. This made the process of producing a “fit” between Lexicon and 
the Consortium difficult. 

Picture-Tel: Two-way Video Conferencing 

Picture Tel is a trade name for two-way video conferencing. The company is the 
Massachusetts Corporation for Educational Telecommunications (MCET). The original idea for 
Picture-Tel use was for a teacher to broadcast instruction from a studio to a class. In 
Massachusetts, on the east coast, as opposed to the west where large distances often separate 
teachers and students, ACLS/Consortium staff suggested that students already have adequate 
access to teachers but could benefit from greater access to each other across sites. A group of 
teachers was invited to visit MCET offices and to explore how Picture-Tel might be used more 
creatively to link students across sites. Circumstances supported creative activity: teachers were 
paid to attend the meeting at MCET and MCET representatives were highly motivated to work with 
the teachers. They did not want the technology to appear to fail. Picture-Tel is now used in the 
Consortium to provide workers with opportunities to talk to each other across sites and to 
collaborate on learning projects. 

This approach to using Picture-Tel involves instructors and workers more thoroughly in the 
design and implementation of a learning opportunity than did the original concept. In this approach 
two teachers collaborate on a lesson plan. Two classes each consider a different aspect of a single 
issue. Through Picture-Tel, they “meet” and contribute to each other’s understanding of the issue. 
For example, two teachers from different sites might plan an “inquiry map” project on health and 
safety. Members of each class choose their own questions to lead the inquiry map but then 
collaborate on the creation of the map with their partner site. A collaboration like this might last for 
three sessions. 

Although instmctors discovered a creative use of the Picture-Tel technology, technical 
problems still make use difficult. It is difficult to install the monitor units and to schedule multiple 
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classes in two sites within the time period allotted. Installation is a “technological nightmare” in 
some areas of the Commonwealth involving transfer problems between analog and digital signals. 
Nonetheless, at the time evaluation data were collected on the Consortium’s distance learning 
component, four program pairs planned to use Picture-Tel. 

ACLS/Consortium staff suggest that the mini-course. Lexicon and two-way video 
conferencing may have found a more comfortable place in the Consortium’s array of services had 
all programs been allowed to develop to a point where staff felt they needed, or, at least, could use 
the technologies creatively. Early demands for use of technology by ACLS/Consortium staff set a 
tense tone with the sites. Implementation could have waited until Year 2, when most sites were 
more ready to explore something new. Creative use of Pipture-Jel did come forward in Year 2, 
delayed from the original start-date by downsizing and staff turnover at MCET. 

■^Conclusions: Goal #2 

* PETS a re functioning as planned but may benefit from further technical assistanee. Overall, 
local PETs are functioning as planned — as the embodiment of partnership at the local level. 
However, in Year 3 they might benefit from additional technical assistance and support from 
the CPET. With the exception of a few companies where worker involvement genuinely 
permeates work organization, conflicts of interest among stakeholders will likely appear. 
Issues like worker participation in PETs, the role of teachers as worker advocates, and the 
PETs role in establishing opportunities for application of skills at work might be addressed in 
state-wide forums or regionally and advance the function of PETs considerably. 

* The CPET has ably guided the Cons o rtium through a complex plan of service delivery to 25 
program s and may now benefit from implementing a formal planning and evaluation proee.ss 
to guide it through a final year. The CPET has served the Consortium well, largely in a 
monitoring role. ACLS/Consortium staff might now encourage CPET members to think 
more proactively about what they can accomplish in Year 3 through facilitating a formal 
planning and evaluation process. The Consortium’s Year 3 emphasis on institutionalization 
suggests a need for additional focus on local evaluation activities and how they might serve 
institutionalization. 

* The Consort ium explored three types of distance learning opportunities and the results of their 
use are still unclear. The Consortium experienced something like “media initiation by fire” 
and emerged with reasonable products and significant learnings about how to proceed with 
future media projects. Actual results for students and programs are unclear at this time It is 
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clear, however, that it is not as easy as it might seem to produce quality education products 
using telecommunications and computer media. Experienced educators need experienced 
media and computer experts to translate good practice into good pictures. While there is no 
doubt that more and more educational products for adults will be produced on video and for 
computers, and it is therefore worthwhile for the Consortium to consider continuing 
production and dissemination, sufficient expertise, time and resources need to be allocated to 
make the development process and products worthwhile. 

• ACLS/Consortium staff learned that there must be a congruence between the technology, 
instructional content and the readiness of teachers and students to work with the technology 
There was agreement among Coordinators and ACLS/Consortium staff that activities in all 
three media projects were premature and would have benefited from more development time. 
As noted below, at the end of Year 2, the Curriculum Working Group is ready to share some 
of the materials and formats that it has been compiling over a two year period. It took two 
years for the group to identify all the dimensions of its task and come to agreement about how 
to execute that task - documentation and dissemination of workplace curricula. Had the group 
been pressured to produce documentation in Year 1, it is unlikely that its format and content 
would be as rich as they are now. 



^Goal #3: Promote the institutionalization of workplace literacy 

programs at the twenty-seven businesses involved in the Consortium. 
Indicators: 

• Consortium supports discussion and planning for institutionalization through trainings and 
other means 

• Availability through the Consortium of models of institutionalization developed in formerly 
funded programs to guide current efforts 

^Evaluation questions: 

• What trainings or other supports has the Consortium provided to guide the institutionalization 
process, including models of institutionalization developed in formerly funded programs to 
guide current efforts? • Which programs will continue beyond federal funding? 



As the Consortium entered the second quarter of Year 2, CPET members became more 
acutely aware of the need to address if and how programs will continue beyond federal funding. 
ACLS/Consortium staff convened a panel on institutionalization at the February 1996 CPET 
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meeting, and that panel set the tone for how CPET members would address institutionalization 
from that point on. Corporate Executives whose companies institutionalized their programs after 
federal funding was cut in earlier funding rounds of the National Workplace Literacy Program 
joined ACLS/Consortium staff on a panel that described how their companies made the decision to 
continue their programs. CPET members clarified the current status of institutionalization in their 
organizations and identified barriers and questions to be addressed both in their local programs and 
at the DOE. 

The panel discussion helped CPET members to understand that greater program visibility 
might help to insure continued funding. Hearing advice on how to get good public relations for 
your program — for example, becoming involved with Picture-Tel or presenting at a Network 
conference — from committed business people rather than from other program personnel helped 
CPET members to place institutionalization and dissemination higher on their agendas. It also 
helped to make CPET members feel that they were up to the job. Apparently, some CPET 
members sometimes find conversations about institutionalization with company representatives 
self-promoting — more like looking to sell a service than meet a need. ACLS/Consortium staff 
might continue to coach CPET members on this point. 

Institutionalization becomes a more-meaningful issue as programs face shutdown. 
Evaluation also takes on greater meaning because some companies might want evidence of 
program accomplishments before committing to continued funding. A central message of the 
evaluation process which CPET members supported in their local PETs, is: consider what 
evidence you will need to persuade all your stakeholder audiences about the value of your 
program. It is doubtful that the urgency of that message was conveyed in the first program year, 
but the CPET can be supported to convey that urgency to local PETs at the start of Year 3 and to 
translate urgency into actual evaluation activity. 

How many Consortium programs are likely to continue beyond federal funding? The 
numbers in Table 15 were calculated in April 1996. Eight programs (32 percent) said they would 
continue, three of them on a smaller scale. Eight said they didn’t know if they would continue. 
Two said they would not. There is missing data for seven programs. The standard that the 
Consortium set for institutionalization is that 80% of currently funded programs would continue to 
deliver some form of education service. The Consortium must work to insure a reasonable rate of 
institutionalization using this standard. 



48 



Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page 38 



Table 15. Continuation by Type of Industry 





Type of Industry 


Program will Continue 


Education 


Health Care 


Manufacturing 


Yes 


1 (4)* 


1(4) 


3(12) 


Yes, but on smaller scale 




1(4) 


2(8) 


No 




1(4) 


1(4) 


Don’t know 




5(20) 


3(12) 


Missing Data 




3(12) 


4(16) 


♦Twenty-five respondents completed protocols in which questions on Continuation appeared. Cells show 
the numbers and, in parentheses percent of response of the total 25. 



Conclusion: Goal #3 

* CPET members have been pursuing continuation of their programs but the level of 
commitment to continue programs fas of Mav 1996’) falls short of the Consortium standard . 
Coordinators should discuss and plan for program continuation more actively. 
ACLS/Consortium staff might troubleshoot any problems that Coordinators have with 
discussing or planning continuation by including the topic on CPET meeting agendas for the 
next several months. 



■^Goal #4: Strengthen the Consortium effort through the use of 

an independent, external evaluation of the Consortium and its 
individual partnerships. 

This report stands as evidence of the scope of work that the evaluators have accomplished 
in Year 2 of the Consortium. Some of our reflections on the evaluation process are included in 
the Methodology section of this report. We are confident that the process of collecting data 
during site visits, soliciting self-score data from Coordinators, participating in CPET meetings, 
advising on methods of gathering valid outcome data by site, and sharing evaluation results 
have strengthened the Consortium effort. 
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"^Goal #5: Produce and disseminate information on exemplary 

workplace partnerships, delivery models, curricula, and industry- 
specific literacy materials. 

Indicators: 

• Consortium oversees the systematic development of local program materials on exemplary 
workplace partnerships, delivery models, curricula, and industry-specific literacy materials. 

• There is a process or guidelines available to local staff to help them document their curricula 
and other activities. 

• Consortium collaborates with SABES to produce/disseminate materials. 

■^Evaluation Questions 

• Is the Consortium overseeing the systematic development of local program materials on 
exemplary workplace partnerships, delivery models, curricula, and industry-specific literacy 
materials? • Is there a process or guidelines available to local staff to help them document their 
curricula and other activities? • How is the Consortium overseeing this effort? • What 
materials will be produced? • By whom? • What is the dissemination strategy? • Who is 
intended to use the materials inside and outside of the Consortium? • Is the Consortium 
collaborating with SABES on this goal? • In what ways? • To what end? 

The Curriculum Working Group (CWG) 

Unless otherwise stated, data for this section of the report were taken from: 

• the Indicators PLUS protocols completed by Site Coordinators, and other related 
protocols completed by teachers and business and union representatives 

• interviews with ACLS/Consortium staff 

• documentation of CWG meetings 

• materials produced by the CWG 

• the text of the original proposal submitted by the Massachusetts Department of Education 
to the National Workplace Literacy Program. 

The CWG is overseeing the systematic development of local program materials on 
exemplary workplace partnerships, delivery models, curricula, and industry-specific literacy 
materials. Its members have established the process and guidelines for local staff to document 
their curricula in ways that will maximize its usefulness to other workplace educators. The work 
of this group may well account for the overall high ratings which both Program Coordinators and 
the evaluators gave to curriculum-related Quality Indicators. It is certainly a group where 
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imaginative work abounds and where the mission of the Consortium is continually reinterpreted 
and advanced. It is also a group whose efforts will make the Consortium’s accomplishments 
accessible to large audiences. 

The CWG is composed of nine people, at least one from each partnership, with a second 
member from a primarily health care partnership and from the higher education partnership. 
Members were chosen by their willingness to attend the group and do the work. DOE/ACLS 
decided to compensate CWG members for one day a month for time spent on the task. This did 
not fully compensate the members for all their time on the project. 

Early in 1996 the CWG decided to form subcommittees in order to facilitate review of 
curriculum documents by industry. The subcommittee are health care, manufacturing and 
education. The CWG meets each month, alternating between the full group and subcommittees. 

Charge 

The CWG is charged to read all site-based curriculum documents from Year 1 and Year 2 
and write two synthesis documents. Thus, the group is extrapolating major curriculum topics and 
issues from a vast source of curriculum materials — there were 138 courses offered in Period 3 
alone. One document will focus on issues by industry; another will identify more universal issues. 
The group finds it difficult sometimes to distinguish between what is an industry issue and what is 
a universal issue. Much useful discussion in the group revolves around where an issue belongs. 

The CWG is interested in understanding how a teacher thinks about what she is teaching 
and how she makes decisions about what to teach, not just in surface documentation. 
Consequently, curriculum documentation is considered to be ‘data’ and the charge of the CWG is 
to interpret the data — to make thematic sense of it, and present it in a format that is useful to 
workplace educators. The CWG designed a format for curriculum documentation by teachers that 
would keep paper work to a minimum and make the group members’ job of interpretation 
somewhat easier. There are six topic headings, including: description of teaching context; 
description of curriculum development process; challenges/things you would do differently; a 
sample of teaching activities; list of topics covered in the cycle; and a resource list of published 
materials that were useful. See Appendix H: Revised Guidelines for Curriculum Documents. 

Teachers are not required to use the suggested documentation format. However, CWG 
members found that those who followed the guidelines gave the group the most valuable 
information. This was a validation of the guidelines that affirmed the documentation process as a 
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whole. The process has served a staff development function as well. Teacher/C WG members 
have become leaders at local sites. 

Status of Work 

At the time this evaluation report was written, the CWG plan of work was as follows. In 
December 1996, CWG members trade feedback on a draft document with Year 1 examples. By 
mid-February 1997, draft documents with Year 2 examples are disseminated. By mid-April, the 
final narrative is circulated for comment. 

CWG members seek specific examples of issues that are important to an industry. For 
example, programs in health care settings need to address the effects of hospital mergers on work, 
patient care, cost of care and so on. If such an example is not forthcoming from teacher 
documentation, then the CWG will put out a call for the example. This means that the teachers are 
doing significantly more than just compiling sites’ work. They are pulling together data, analyzing 
it, presenting it and bridging back to the Consortium with it. CWG members note that “the 
absence of data is a presence.” They want to address the question: Why are some topics 
emphasized and others not? They are confident that thinking in these terms will insure a much 
more interesting and useful document. 

The Consortium is also using CWG products as a bridge to other teaching issues. For 
example, group members are compiling one page summaries like “promising practices” in which 
the “cream of the curriculum crop” is passed on to all teachers in the Consortium. This builds the 
capacity of the Consortium in a substantial way. 

Process. Product and Dissemination 

The Assistant Coordinator of the Consortium organized a set of activities that stimulated 
members to discuss what a curriculum is; how they could structure curriculum documentation 
(their data) in such a way that would allow comparison, analysis and drawing conclusions; and 
what you would you want to know from teachers before you, if you were a new teacher coming 
into the Consortium. The CWG brainstormed guidelines and brought them back to the teaching 
staff who, in turn, gave them feedback. 

CWG written products are likely to be enormously useful to workplace educator audiences 
within and without the Consortium. It could also be said that the process that the CWG used to 
develop its products is also a product — something that can guide curriculum documentation in 
other workplace education programs and other education venues, adult or otherwise. The CWG 
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wants to connect its work to the curriculum frameworks that are being revisited for adult education 
and that need workplace references. 

The CWG will send information to educators and others on local and national workplace 
education mailing lists and present its findings on MCET shows. It will continue to identify its 
audiences and insure that its materials are disseminated appropriately. The dissemination effort 
might be quite large. Proper dissemination will probably require a support staff person or at least 
dedicated time of a staff person who understands the dissemination task. 

■^Conclusion: Goal #5 

• The CWG is building the capacity of the Consortium at the same time that it is producing and 
disseminating valuable documentation of curricula . The CWG’s written products will likely be 
enormously useful to various audiences. The process the CWG used to document the curricula 
is also a product which may have many applications and a wide audience. ACLS, the DOE and 
the Consortium would be well-served if staff time were allocated to help with dissemination. 



Evaluatioif Objective #2: Determine the leyep:^ 

Indicators of Quality and the relationship of the Indicators to learner diitcdnies, 



work^lac^utcomes^n^^artner^^ 
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Overall Level of Implementation of the Quality Indicators 

We have determined the Quality Indicators' levels of implementation for the 25 sites that 
provided us with completed Indicators PLUS protocols. As described earlier, the evaluators each 
independently rated each site on each Indicator using the completed protocols as the information 
base; we then compared our ratings. Refer to the Methodology section of this report for a more 
detailed description of how the Indicators were scored. 

The mean ratings for each Indicator, shown in Table 16, are generally high, with 18 of the 
25 Indicators analyzed^ being above 5 on the scale of "1" (not implemented) to "6" (fully 
implemented). These ratings are consistent with our impressions from the sample of seven sites 



5 We did not analyze the three Indicators in category 4, Supplementary Services, since only 
five sites reported offering any. We are not certain of the meaning of Supplementary Services for 
those who did and did not report them. Some may offer the services informally and thus did not 
report them. Given both the small number reporting and the questionable meaning of the 
Indicators and associated protocol questions, reports of data in this category would be misleading. 
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Table 16. Summary Statistics of Quality Indicator Ratings Across All Sites 



Indicator 


Mean 


Std Dev 


Minimum 


Maximum 


N 


A 


4.46 


1.61 


2 


6 


25 


B 


5.47 


0.53 


4.5 


6 


25 


C 


5.31 


0.75 


3 


6 


25 


D 


5.3 


0.91 


3 


6 


25 


E 


5.05 


1.38 


1 


6 


23 


i. Partnership and Planning 
Mean 


5.13 


0.8 


2.9 


6 


25 


A 


5.51 


0.66 


4 


6 


25 


B 


5.55 


0.48 


4.75 


6 


25 


C 


5.35 


0.77 


3.5 


6 


25 


D 


5.79 


0.37 


4.75 


6 


25 


E 


4.73 


1.12 


1 


6 


25 


F 


5.64 


0.74 


3.75 


6 


25 


G 


5.2 


1.11 


3.5 


6 


25 i 


n. Curriculum Mean 


5.4 


0.43 


4.29 


5.96 


25 


A 


5.6 


1.02 


1 


6 


25 


B 


3.94 


0.93 


1.5 


5.5 


25 


C 


3.51 


1.07 


1 


4.75 


25 


D 


4.34 


1.28 


1 


6 


25 


E 


4.59 


1.38 


1 


6 


25 


F 


4.29 


1.32 


1 


6 


24 


III. Assessment, Evaluation 
and Outcomes Mean 


4.39 


0.86 


1.83 


5.29 


25 


A 


5.56 


0.45 


5 


6 


25 


B 


5.42 


0.63 


3.5 


6 


25 


C 


5.51 


0.46 


4.75 


6 


23 


V. Staff Mean 


5.48 


0.47 


4.25 


6 


25 


A 


5.24 


0.56 


4.25 


6 


21 


B 


5.15 


0.87 


3 


6 


24 


C 


5.34 


0.95 


2.5 


6 


25 


D 


5.47 


0.75 


3 


6 


24 


VI. Administration Mean 


5.33 


0.61 


3.88 


6 


25 


GRAND MEAN 


5.1 


0.54 


4.06 


5.71 


25 
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we visited where implementation of the Indicators was quite high. The high scores are also not 
surprising since site personnel in these projects tend to be well-experienced with workplace 
education, with the DOE's evolving mode of conducting such projects, and many of the people and 
participating projects were involved either directly or indirectly in the development of the 
Indicators. 

Across the six major content areas of the Indicators, the means for category 5, Staff, are 
consistently the highest. Very close to Staff in high quality ratings are the categories of Curriculum 
and of Administration. These ratings suggest that the quality of the staff and the support activities 
for them, the quality of the curriculum development process and its products, and the 
administrative features of policies and resources for program support, program coordination 
activities and personnel, and the incentives for worker participation are particular strengths within 
the Consortium. 

Category 3, Assessment, Evaluation and Outcomes, is notably weak. At the time these 
data were collected, few sites had collected, developed, or implemented the assessment and 
evaluation procedures necessary to conduct a sound program evaluation. Programs usually had in 
place good individual student assessments, which were used informally for formative instructional 
purposes, but those data were not in a form that could be synthesized to make evaluative statements 
about program outcomes. In some cases the data existed in appropriate form, but had not been 
analyzed at the class or program levels. 

By Business Context Characteristics 

A partnership or site implements the features covered in the Quality Indicators in the context 
of the business that is part of the partnership. It is not unreasonable to expect that some of those 
companies characteristics might influence the degree to which some of the Indicators can be 
implemented. We therefore examined the relationships between levels of Quality Indicators' 
implementation and business type, size, and percent of unionized employees involved in the 
business. Since the numbers in each breakdown category of the business variables are fairly small 
and the Quality Indicator data are far from normally distributed, we tested the differences across 
subgroups with non-parametric statistics for each. The median ratings for each Indicator by 
business type are shown in Table 17. 

Note that there is a pattern of lower medians for the manufacturing sites than for the health 
sites. Among those are a substantial number of differences between the business types, on 16 of 
the 25 Indicators. This is a striking result and one which raises the question of how to account for 
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Table 17. Median Ratings for Quality Indicators by Business Type 





Business Type 


Indicator 


Health 


Manufacturing 


(N=12) 


(N=ll) 


la 


1 5.75 


2 


lb 


6 


5 


Ic 


6 


5 


Id 


5.88 


5.5 


le 


5.5 


5.5 


Partnership & Planning Total 


5.88 


5.5 


2a 


5.75 


- 6 


2b 


6 


5 


2c 


5.5 


6 


2d 


6 


6 


2e 


5.25 


4.5 


2f 


6 


6 


2g 


I 6 


i 3.5 


Curriculum Total 


6 


6 


3a 


6 


5.5 


3b 


4.63 


3.5 


3c 


4.38 


3.25 


3d 


5 


3 


■=3e 


5.5 


3 


3f 


5 


4 


Evaluation Total 


5 


3.13 


5a 


6 


5 


5b 


5.88 


5 


5c 


5.88 


5 


Staff Total 


5.88 


5 


6a 


5.25 


5 


6b 


5.75 


5 


6c 


6 


5 


6d| 


5.88 I 


5.75 


Administration Total 


5.82 


5 


Grand Total 


5.88 


5 
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the differences. One possibility is a partnership effect. That is, the partnerships work with either 
manufacturing sites or health sector sites exclusively (except for one partnership that has four 
health sites and one manufacturing site). Since a lot of the activities covered in the Quality 
Indicators are mostly in the control of the education partner common to a number of business 
partners, then the extent to which the educational partner implements the Indicators will be reflected 
in the Quality Indicator scores of each of its constituent sites. Thus, it may be that those education 
partners with health sector business partners are more effective in implementing the Indicators. On 
the other hand, manufacturing companies may present more difficult barriers to implementing the 
"ideal" factors included in the Indicators, somewhat beyond the control of the educational partners, 
and these may be reflected in their systematically lower Quality Indicator ratings. If the latter 
hypothesis is true, then further study may be useful to discern what those greater difficulties are. 

We explored the differences in Quality Indicator ratings according to business size and 
percent of union membership at the company and found only minor differences. 



Relationship of Learner Outcomes to Quality Indicators 

One of the central assumptions underlying the Quality Indicators is that their 
implementation is related to workers' learning outcomes. The first opportunity to test that 
assumption lies with the Period 3 data, in which we correlated the Indicator ratings with the gains 
on each of the seven learner outcome areas defined on the Learner Form, with the site as the unit of 
analysis. Because of the generally low correlations and the size of the correlation matrix (3 1x7), 
we do not present them in a table. The correlations were very low and the number of significant 
correlations did not exceed what we would expect by chance alone. 



We know that in Period 3 instructors used the following assessment procedures: group or 
one-on-one interviews in 68 classes (87 percent); “other” in 45 classes (58 percent); portfolio 
assessment in 37 classes (47 percent); individual learning or educational plan in three classes (38 
percent); and learner work examples in 29 classes (37 percent). Standardized tests were used in 
only four classes (5 percent). Customized, job-related skills competency tests were used in only 
three classes (4 percent). This means that assessment procedures are varied and — to the best of 
our knowledge — no systematic attempt has been made to devise a scoring procedure that would 
enable instructors to translate an interview or portfolio assessment into a numeric representation of 
a learning gain. We are aware that this is an issue not only in workplace education but in all of 
adult education. If there are to be quantitative evaluations of programs that examine relationships 
between learner outcomes and other variables, then outcome data need to be put in meaningful 
quantitative form. This is an issue that the Curriculum Working Group might examine. 
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Relationship of Quality Indicators to Workplace Outcomes 

The other central assumption about the Quality Indicators is that they are related to 
workplace outcomes. We examined that assumption with the self-reported workplace outcomes in 
the three outcome categories for which more than five sites reported the outcome, namely, 
improved communications, less scrap and rework, and improved participation. We then compared 
the mean Quality Indicator ratings for the sites with and without each outcome. There were no 
differences in Quality Indicator ratings for sites that reported improved communications and those 
that did not report them as workplace outcomes of the project. There are differences on four of the 
indicators between those sites with and without reports of reduced scrap and rework. Those sites 
that did report reduced scrap and rework had lower Quality Indicator means. Those that did not 
report it had higher Quality Indicator means. There was also one difference for the improved 
participation outcome, that on the Indicator for evaluation data used for program and policy 
decisions. The other Quality Indicator means showed no pattern of favoring any outcome's 
presence or absence. 

We also created a variable that was the number of categories on which the site reported 
some positive outcome. As described earlier, these values ranged from one to four, and we 
compared the means of the sites with one, two, three, and four outcomes. There was no pattern of 
differences across the sites. 

Relationship of Quality Indicators to Partnership Outcomes 

In Period 3, commitment to continue a program after federal funding ceases is considered a 
partnership outcome. While there may be reasons that a partnership does not continue after federal 
funding ceases that are not related to the quality of the partnership per se (partners may have 
enjoyed a successful collaboration but the program may not continue), continuation is nonetheless 
a key partnership outcome that the Consortium has promoted. 

We explored the extent to which the sites' reports of their intentions and plans to continue 
the program after Federal funding ended was related to their Quality Indicator ratings. Sites' 
responses to open-ended questions on the Indicators PLUS Protocol were content-analyzed and 
classified as: “No, do not plan to continue”; “Don't know”; “Yes, plan to continue on a smaller 
scale”; “Yes.” The median Quality Indicator ratings for sites within each category are shown in 
Table 18. There is a pattern in the results, namely that the sites that responded "Yes, but on a 
smaller scale" had lower median ratings than the other three continuation groups on Quality 
Indicator categories 1 (Partnership and Planning), 3 (Assessment, Evaluation and Outcomes), 5 
(Staff), and 6 (Administration). The other continuation groups seem to vary in how they rank on 
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Table 18 . Median Quality Indicator Ratings by Continuation Status 



Quality Indicator 


Will the program continue'/ 

Don't Yes, 

know smaller 

No (N=2) (N=7) scale (N=3) Yes (N=5) 


lA 


5.75 


5.5 


2 


4.75 


IB 


5.38 


6 


5 


6 


1C 


5.5 


5.5 


5 


5 


ID 


5.75 


6 


5 


5.5 


IE 


5 


5.25 


6 


5.25 


1 . Planning & Partnership Mean 


5.58 


5.65 


4.6 


5.5 


2A 


4.75 


5.5 


6 


6 


2B 


5.63 


6 


5 


6 


2C 


4.5 


5.5 


6 


6 


2D 


5.13 


6 


6 


6 


2E 


4 


5 


4.5 


5 


2F 


5 


6 


6 


6 


2G 


3.75 


6 


3.5 


6 


2. Curriculum Mean 


4.68 


5.57 


5.29 


5.57 


3A 


5.5 


6 


5.5 


5.5 


3B 


4.25 


5 


3.5 


4 


3C 


3.63 


4.5 


3 


4 


3D 


3.88 


5 


3 


5 


3E 


5.63 


5.5 


3 


5 


3F 


4.5 


4.75 


4 


4.88 


3. Evaluation Mean 


4.56 


5.13 


3.75 


4.83 


5A 


5.63 


5.75 


5 


6 


5B 


5.63 


6 


5 


5.5 


5C 


5.38 


5.75 


5 


5.5 


5. Staff Mean 


5.54 


5.83 


5 


5.67 


6A 


5 


5 


5 


5.25 


6B 


5.25 


5.88 


5 


5.25 


6C 


5.75 


6 


5 


6 


6D 


5.5 


6 


4.5 


6 


6. Administration Mean 


5.5 


5.81 


4.83 


5.56 


GRAND MEAN 


5.05 


5.52 


4.65 


5.49 




59 

Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page 49 



Quality Indicator ratings from one category to another. While this leads us to no clear conclusions, 
perhaps more refined data on this question will yield a better answer. 

"^Conclusions: Evaluation Objective #2. 



« The level of implementation of the Quality Indicators as scored in Year 2 is quite high. 

The sites and partners have implemented the Quality Indicators at a very high level. The average 
ratings on 18 of the 25 Indicators (72 percent) were above 5 on a 6-point scale (with 6 as the 
highest level of implementation). The strongest areas were Staff, Curriculum, and Administration; 
the weakest was Assessment, Evaluation and Outcomes. Additionally, the level of Quality 
Indicator implementation was significantly higher for health sites than for manufacturing sites on 
64 percent (16 of 25) of the Indicators, a result that is possibly explained as a partnership effect. 
Alternately, if manufacturing companies present more difficult barriers to implementing the "ideal" 
factors included in the Indicators, then further study would be required to discern what those 
greater difficulties are. 

« The relationship between Quality Indicator Implementation to Worker, Workplace, and 
Partnership Outcomes is not clear at this time . A central focus of the Consortium’s Year 2 
Evaluation was testing the assumption that there is a relationship between the presence of Quality 
Indicators and positive outcomes at the learner, workplace, and partnership levels. We did not find 
a strong relationship. Correlations between self-reported learning gains and sites' Quality Indicator 
ratings were generally low and there were few differences between sites' Quality Indicator ratings 
for those that did and did not report workplace outcomes. Those few correlations that were high 
did not seem to fit any logical pattern or explanation. 



• With modification the Quality Indicators can be useful for further research and evaluation. 
The Quality Indicators have undisputed benefits when used as a guide for program development. 
The effort to test their validity by developing a scoring method for them and correlating those 
scores with outcomes is potentially a useful research focus, one that we pursued this year but 
without a clear conclusion. There were several reasons for this: 

(1) Efforts to create a scoring system based on performance standards for the Quality 
Indicators is underway and may eventually lead to better opportunities to study the relationship 
between them and outcomes. 

(2) The outcome data with which we worked was scant at best. Testing the relationship 
between Quality Indicators and outcomes relies as much on good outcome data as it does on a 
satisfactory scoring system for the Quality Indicators. Scant outcome data, a by-product of the 
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unexpected dearth of NWLIS data, as well as limited anecdotal reporting from local sites, 
hampered our analysis of the Quality Indicators as much as any difficulty in the Quality Indicator 
scoring process. 

Continued work on the relationship between Quality Indicators and outcomes will likely 
prove useful to many audiences. Refined performance standards for the Quality Indicators can 
serve multiple purposes, including guiding programs in their development, possibly through a self- 
scoring process. Improved documentation of outcomes will also serve many audiences, among 
them local PETs who can use this information to argue for program continuation. The resources 
and time needed to continue the proper study of the relationship between Quality Indicators and 
outcomes, however, are at this time quite extensive. The DOE may be well served to continue this 
work internally with evaluator support and turn the focus of the Consortium evaluation onto 
questions of more immediate concern and impact, including, as mentioned, how PETS are 
functioning and the benefits of the Consortium structure. 



Evaluation Objective #3: Determine the :;re^^ instructional 

mettiodo^ies and worker and workplace obtcQnieis. ^^^^^ 



Learning Gains and Instructional Characteristics 

We examined the relationships between workers’ self-reported gains in seven learning 
areas and the following instructional characteristics: number of students in the course, number of 
course hours, type of service, and primary course emphasis. These are the instructional 
characteristics on which the programs varied and over which they might have some control. We 
examined the four characteristics in relation to the gain scores for all students who reported for 
those courses. Though these appear to be the best data available to examine these relationships, 
there are some limitations which qualify the results and which therefore bear mentioning. Of the 
138 courses reported offered by the Consortium during Period 3, only 56 had at least some 
individual students report their pre- and post- scores from which we could calculate the gains. 
Thus, these results are based on about 4 1 percent of the total courses offered. As noted before, of 
the 534 students reported to have been served during the period, only 156 - 181 have gain scores 
for the individual analyses conducted. Given these limitations, the findings are as follows. 
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Course Length 

Table 19 presents the seven content areas and the sum of their scores. For course length, 
there is a pattern that the average gains in three of the seven content areas (three of the four 

Table 19. Mean Gain Scores by Course Length 





Length of Course* 


Gain in (N = # students with scores) 


24-52 Hours 
(N=44 courses) 


54-76 

Hours 

(N=44 courses) 


78-120 

Hours 

(N=46 courses) 


Reading English (N=179) 


0.03 


0.13 


0.15 


Understanding English (N=174) 


0.02 


0.28 


0.13 


Speaking English (N=174) 


-0.01 


0.41 


0.13 


Writing English (N=173) 


0.03 


0.43 


0.21 


Working as a Team (N=168) 


-0.03 


0.03 


0.04 


Using Math (N=160) 


0.13 


-0.06 


0.09 


Solving Problems/Reasoning (N=156) 


0.18 


-0.03 


0.11 


Sum of Gains 


0.33 


1.22 


0.86 



* The courses were simply divided into the three most equal thirds possible for this analysis. 



English-related ones) and the sums were highest for the mid-length courses, those between 54 and 
76 hours. Moreover, two of the differences, in speaking and writing English, were especially 
pronounced. In the same three content areas, the longest courses, between 78-120 hours, had the 
next highest gains and the shortest courses had the lowest gains. The remaining English-related 
area, Reading English, had the highest gains for the longest courses and the next highest for the 
mid-length courses, though nearly the same as each other. This pattern did not hold for the math 
or problem-solving outcome areas, where the shortest courses had the highest gains. Even though 
the apparent relationship between course length and student self-reported outcomes is not linear, 
we explored the correlations between course length and each of the seven gain measures. All 
correlations were very low, between +.19 and -.07, likely reflecting the non-linear relationship 
between the variables. This finding suggests that short courses are least effective for at least a 
substantial proportion of the students served in the Consortium and in courses for limited-English 
proficient adult workers in ESOL courses. 
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Number of Students in a Course 

We next examined whether number of students in the course was related to the gains. The 
range of student numbers in courses was fairly narrow, from two through 19, with 50 percent of 
the courses having nine or fewer students and the other half having from 10 to 19 students. 
Therefore we split the entire sample into those two groups and compared the gain scores in the 
same manner as above. The results are shown in Table 20. There appears to be no relationship 

Table 20. Mean Gain Scores by Number of Students in Course 





Number of Students in 
Course* 


Gain in (N= # students with scores) 


2-9 

(N=67 

courses) 


10- 19 
(N=67 
courses) 


Reading English (N=181) 


0 


0.14 


Understanding English (N=176) 


0.11 


0.1 


Speaking English (N=176) 


0.06 


0.13 


Writing English (N=175) 


0.08 


0.2 


Working as a Team (N=170) 


0 


0 


Using Math (N=162) 


0.04 


0.11 


Solving Problems/Reasoning (N=158) 


0.21 


0.07 


Sum of Gains 


0.49 


0.78 


*The courses were simply divided into the two most equal halves possible for 
this analysis. No differences between these two groups were statistically 
significant. 



between the number of students in the course and their self-reported gains. Three of the 
differences favor the smaller groups, three the larger groups, and one has no difference. The 
differences, moreover, are generally small. The overall sum of gains is slightly higher for the 
larger classes, but not high enough to suggest a strong difference between the groups. 



Type of Service Offered 

The type of service offered might be expected to be closely related to student gains that 
match the services. For example, we would expect English gains to result from ESOL services 
more than from other types of services. This expectation is tested with the data in Table 21. The 
data show that all four English-related gains and problem-solving gains were the largest for those 
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in Beginning and Intermediate ESOL classes. Math was the strongest gain area for the pre-GED 
classes. All English-related areas and teamwork showed the weakest gains in pre-GED courses. 
The clear pattern of English classes showing higher gains for ESOL classes tends to suggest 
validity of the measures, for this is to some degree the result we expect. The pre-GED courses' 
generally weak gains may be attributable to their relative emphasis on math and not 

Table 21. Mean Gain Scores by Type of Service 



Gain in (N= # students with scores) 


ABE 


T 

Pre-GED 


ype of 
GED 


Service 
Beg. ESOL 


Int. ESOL 


Reading English (N=181) 


0 


-0.02 


0 


0.15 


0.13 


Understanding English (N=176) 


0 


-0.1 


0 


0.08 


0.29 


Speaking English (N=176) 


0 


-0.2 


0 


0.18 


0.3 


Writing English (N=175) 


0 


-0.1 


0 


0.22 


0.34 


Working as a Team (N=170) 


0 


-0.02 


0 


0 


0.02 


Using Math (N=162) 


0 


0.15 


0 


0.13 


0.02 


Solving Problems/Reasoning (N=158) 


0 


0.1 


-0.17 


0.33 


0 


Sum of Gains 


0 


-0.03 


-.16 


1.08 


1.13 



English usage compared to the ESOL classes, a feature that could be investigated in the future for a 
more conclusive result. 

Primary Course Emphasis 

We also investigated the relationship of the primary emphasis of the course to the gains. 
Primary course emphasis overlaps somewhat with type of service, but some courses with the same 
service (e.g., ESOL) emphasize reading while others emphasize writing or communications 
(assumed to be oral because of the way it is listed on the DOE Course Data Form). Using the same 
format as above, the results are shown in Table 22. Courses with their primary emphasis on 
Reading demonstrate the highest gains on the sum of gains and in all the English-related areas, 
even those of speaking and understanding English. Curiously, the courses designated by the 
respondents as having a primary emphasis on Communications do not make good gains, even in 
the oral communications areas of speaking and understanding. Those courses with emphasis on 



This result provides some, though far from thorough, evidence of the content validity of 
the learner self-report scores. 
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both Reading and Math show all negative "gains" in the English-related areas and in Working as a 
Team, but show positive gains in Math and Problem-solving, albeit not strong gains in the latter. 
The clearest pattern in the results, that of the Reading emphasis courses, suggests that an emphasis 
on Reading is the most efficacious means of improving all English-related communications skills, 
whether written or oral. Given the limitations of the data, however, as described earlier, this 
finding warrants further inquiry rather than any implications for practice at this time. 



Table 22. Mean Gain Scores by Course Emphasis 



Gain in 

(N= # students with scores) 


Literacy 


C 

Reading 


ourse Er 
Reading 
&Math 


nphasis 

Writing 


Comm's 


Other 


Reading English (N=181) 


0.09 


0.21 


-0.05 


0.19 


-0.11 


0.19 


Understanding English (N=176) 


0.08 


0.35 


-0.14 


0.05 


0.07 


0.19 


Speaking English (N=176) 


0.13 


0.5 


-0.18 


0 


0.04 


0.11 


Writing English (N=175) 


0.18 


0.58 


-0.14 


0.19 


-0.04 


0.15 


Working as a Team (N=170) 


-0.06 


0 


-0.05 


0.25 


-0.04 


0 


Using Math (N=162) 


0.16 


-0.13 


0.14 


0.28 


0 


0.04 


Solving Problems/Reasoning (N=158) 


0.26 


0 


0.05 


0.11 


0 


0.12 


Sum of Gains 


.77 


1.10 


-.40 


1.06 


-.08 


0.81 



Relationship of Instructional Characteristics to Workplace Outcomes 

Crosstabulations of type of service and primary course emphasis on the sites' self-reported 
workplace outcomes, using the same categories as in Table 10, Frequencies of Workplace 
Outcomes, showed no discernible relationships. Comparing the mean number of course hours, 
however, between those sites reporting outcomes in the three most frequently reported categories 
and those not reporting any in those categories yielded the results shown in Table 23. Those sites 
that reported outcomes in reduced scrap and rework and in improved participation had courses with 
significantly fewer hours' length than those who did not report those areas as outcomes. Those 
that reported improved communications outcomes had courses slightly longer than those who did 
not report outcomes in this category, but the difference was not significant. 
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Table 23. Mean Course Hours by Workplace Outcomes Reported 





Reported as 
Outcome 


Outcome Category 


Yes 


No 


Improved communications 


67.1 


64.9 


Reduced scrap & rework 


52.6 


74.9 


Improved participation 


58.4 


77.9 



Relationship of Business Characteristics to Worker Outcomes 

We explored the hypotheses that different business characteristics might be related to 
workers’ gains, using the key business characteristics and the self-reported workers’ gains 
analyzed earlier. 

Type of Business or Industry 

Types of businesses or industries and the gain scores for students in each are shown in 
Table 24. The means use the learner as the unit of analysis; that is, they are computed from all 
158-181 individual scores depending on the gain area. The results show a trend of manufacturing 
sites attaining higher gains on the sum of gain scores and in the English usage areas, with health 
sites next, and the university site with the least gains. Those differences are most pronounced for 
two areas: speaking and writing English. This is similar to what occurred when looking at the 
relationship between learning gains and instructional characteristics (mid-length courses, 
intermediate ESOL, and course emphasis). When instructional characteristics made a difference in 
learning gains, it was most often in speaking and writing English. 
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Table 24. Mean Gain Scores by Type of Business/Industry 



Gain in (N) 


Education 

(N=l) 


Business T; 
Health 
(N=6) 


ype 

Manufacturing 

(N=6) 


Reading English (181) 


0.04 


0.08 


0.18 


Understanding English (176) 


-0.02 


0.12 


0.23 


Speaking English (176) 


-0.05 


0.11 


0.32 


Writing English (175) 


0.02 


0.14 


0.37 


Working as a Team (170) 


0.04 


-0.03 


0 


Using Math (162) 


0.11 


0.15 


-0.05 


Solving Problems/Reasoning (158) 


0.13 


0.19 


-0.03 


Sum of Gains 


0.26 


0.75 


1.06 



Size of Business 

Considering size of the business, results are shown in Table 25. For the English usage 
areas, the medium-sized businesses have the largest gain scores. There is no clear pattern for the 
ordering of the large or small businesses or for the other outcome areas. The sums of gain scores, 
however, show an ordering that medium sized businesses have the highest average sums, the large 
businesses second, and the small business lowest. 



Table 25. Mean Gain Scores by Size of Business 



Gain in (N) 


E 

Small 

(<250) 


business Si; 

Medium 

(251-500) 


ze 

Large 

(>500) 


Reading English (181) 


0.08 


0.18 


0.07 


Understanding English (176) 


0.13 


0.18 


0.07 


Speaking English (176) 


0 


0.29 


0.07 


Writing English (175) 


0.09 


0.41 


0.08 


Working as a Team (170) 


-0.1 


-0.25 


0.08 


Using Math (162) 


-0.05 


-0.06 


0.09 


Solving Problems/Reasoning 
(158) 


0.11 


-0.14 


0.06 


Sum of Gains 


0.27 


0.66 


0.52 
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■^Conclusions: Evaluation Objective #3. 

The initial emphasis in this objective was on the relationship of instructional characteristics 
to outcomes. We added the relationship of business characteristics to outcomes under this 
objective. Conclusions are offered to stimulate further thinking about possible research and 
evaluation questions rather than on answers . 

• The gains in English usage areas may be related to some instructional and business 
characteristics . The clearest information about learner outcomes is on the four areas of English 
usage: reading, understanding, speaking and writing. The speaking and writing English outcomes 
were always more pronounced than reading and understanding. They were also pronounced when 
we examined their relationship to business and instructional characteristics. 

English usage gains were related to course length, with the mid-length courses (54-76 
hours) showing the highest gains, the lengthiest courses (78-120 hours) showing the next highest 
gains, and the shortest courses (24-52 hours) showing the smallest gains. This suggests that short 
courses are not advised for a predominantly ESOL population. Courses with the primary emphasis 
on reading had the highest gains in all four English usage areas. Beginning and intermediate ESOL 
courses also had the highest gains among the various course content areas queried. There was no 
relationship of gains to class size. Other outcome areas (math, teamwork, and problem- 
solving/reasoning) were not clearly related to instmctional characteristics. 

Among the business characteristics, all three examined were related to gains in the English 
usage outcome areas, and showed no pattern in relation to the other outcome areas. The medium- 
sized businesses had higher gains than the smaller and larger businesses, with the latter two 
alternating on amount of gain across different outcome areas. 
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The following recommendations are divided into two categories: Improve Consortium Function 

and Improve Evaluation Activities 

Improve Consortium Function 

ACLS/Consortium staff should: 

• Support the CPET to become a more formally reflective governing body that is prepared to 
promote institutionalization of programs in the Consortium. In Year 3 of the Consortium, the 
CPET can become more programmatically proactive and less managerial, ACLS/Consortium 
staff should introduce an evaluation process to guide CPET members’ decisions about what they 
want to accomplish between now and the end of the grant period and after. Specifically, CPET 
evaluation activities can enable institutionalization of programs beyond the federal funding 
period. 

• Determine which programs are likely to continue and assess what information and support other 
programs need to enhance the likelihood that they will also continue. With a promise from only a 
third of the programs to continue at this time, actions need to be taken to insure greater 
continuation rates. This may mean providing support to Coordinators who shrink from 
“promoting” their programs. In addition, it might be useful to review the seven program 
expansion and continuation plans in the original grant proposal. 

• Support more evaluation activities in local PETs. A central message of the evaluation process 
which CPET members supported in their local PETs is: consider what evidence you will need to 
persuade all your stakeholder audiences about the value of your program. It is doubtful that the 
urgency of that message is conveyed in the first year or two but the CPET can be supported to 
convey that urgency to local PETs at the start of Year 3 and to translate urgency into actual 
evaluation activity. 

• Consider organizing several state-wide sharings that focus on PET and CPET development.. 
Topics might include: extent of worker participation on PETs; the teacher as student advocate; 
and other issues that might be too difficult for the PET to handle alone. For example, workers 
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might be reluctant to speak up about how organizational structures impede full participation in a 
program. The time seems ripe for supporting PETs and the CPET to grow into the next 
developmental stage. 

• Support the CWG to explore how student outcome data can be expressed in a uniformly 
quantitative way across programs. Any evaluation that examines the relationships between 
student outcomes and other variables must rely oh outcome data that are quantified uniformly 
across programs. Quantifying student outcomes poses a challenge for adult educators who, like 
many in the Consortium, prefer authentic assessment to standardized tests. The CWG may offer 
unique insights into - if not solutions to — that challenge in the workplace setting. 

• Investigate whether the Consortium has leveraged more matching company funds them other 
federally-funded programs. If the Consortium sets a standard for cash matches and stretching 
federal dollars, then it should be aware of this and use the information to promote itself. 

• Develop a dissemination plan for CWG materials; allocate staff time for dissemination; and use 
Consortium materials to promote program continuation. 

Improve Evaluation Activities 



Evaluators should: 

• Work with ACLS/Consortium staff to establish performance standards for Quality Indicators tind 
develop new scoring process for Quality Indicators. 

• Use evaluation results as an agenda for discussion among ACLS/Consortium staff, CPET 
members and local staff and PETs to prioritize interest areas, and develop hypotheses for testing 
with stronger data. 

• Encourage the CPET to discuss evaluation results in CPET meetings and consider how results 
might inform program improvement . 

• Gather more substantive data on how PETs are functioning than the Indicators PLUS protocol 
allowed in Year 2. 



70 



o 

ERIC 



Massachusetts Workplace Literacy Consortium Year 2 Evaluation Report 



Page 60 



• Support the CPET and local PETs to systematically assess both workplace and partnership 
outcomes. This includes supporting Coordinators to determine if data that companies already 
collect might be used to document workplace outcomes and to calculate ROI. 
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INDICATORS OF QUALITY 
FOR WORKPLACE EDUCATION PROGRAMS 



Massachusetts Workplace Literacy Consortium 

July 1995 

1. PARTNERSHIP AND PLANNING 

a. There is a partnership among key stakeholder groups (educators, 
workers, unions, management and supervisors, volunteers, others as 
appropriate) whose representatives meet regularly, are formally 
organized (for example, in a "team,") and employ shared governance of 
the program. 

b. Partners each contribute to defining program goals, have common 
expectations about program activities, and contribute their special 
resources to the program, including time, money, materials, space, 
general knowledge and special expertise. 

c. There is a comprehensive plan for the program that is developed by all 
key stakeholders and reviewed regularly by them. 

d. There is knowledge and support of the program and involvement in its 
planning and governance by upper management, team leaders or 
supervisors, and union leadership. 

e. In business-driven programs, the program is integrated into the 
company's long term plan for organizational development. In labor- 
driven programs, the program is integrated into the union's long term 
plan. 

2. CURRICULUM 

a. There is a process for continually assessing learning needs and 
developing and improving curriculum to meet those needs. 

b. There is a curriculum which is customized (contextualized) to the 

needs of program stakeholders, including: workers, instructional staff, 

management, labor, and others, as appropriate. 

c. Needs assessment, goals, objectives, learning activities, instructional 
activities, evaluation and a feedback mechanism are written or 
otherwise explicit or easily inferred for each unit within the curriculum. 

d. The curriculum incorporates principles of adult learning in an adult 
learning environment and uses adult-oriented materials. For example, 
the curriculum: 



1. employs reinforcement and multiple formats 

2. is contextualized to the life experiences and workplace needs of 
learners 

3. accommodates individual differences in student learning rates and 
styles 

e. Workers are given opportunities to practice and demonstrate abilities 
they are developing in classes. 

f. There is a system for documenting the curriculum and a format for 
dissemination. For example, the curriculum is: 

1. written or videotaped 

2. comprehensive, i.e. covers all major content areas required to 
meet program goals) 

3. described in sufficient detail to guide potential users or adapters 

g. Instruction takes place in a physically supportive environment (e.g., 
well lighted and ventilated; minimal noise; comfortable seats and 
writing surfaces) 

3. ASSESSMENT, EVALUATION AND OUTCOMES 

a. There are clearly articulated goals for the program which may evolve 
over time. 

b. There is a procedure in place to assess anticipated and unanticipated 
progress over time in three areas: 

1. the progress of participants (which may include sub-categories; for 

example: the progress of participants as adult learners, workers, 

and community members 

2. the impact of the program on the workplace 

3. the quality of the partnership 

c. There is evidence of a broad base of results, including impact on 
learners, the workplace, and the program partnership. The results 
demonstrate whether anticipated goals were achieved and also 
adequately describe unanticipated outcomes. 

d. There is a rationale for the types of data collected. 

e. Data are reliable, rapidly accessible, updated regularly, and protect 
the rights of individuals. 

f. Data are used for program development, policy decisions, and for 
internal and external communications. 
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4. 



SERVICES 



a. There are supplementary learner services provided which are 
appropriate for the needs of the program's population, including 
educational counseling, childcare, and transportation. 

b. Learners are aware of the availability of supplementary services and 
there is no stigma or threat associated with their use. 

c. Staff support use of supplementary services and integrate their use 
into daily program operations. 

5. STAFF 

a. Staff are competent to teach adults in job-related workplace settings. 
Competence derives from training, experience or personal 
characteristics. 

b. Staff are oriented to the workplace and are provided with ongoing 
opportunities for training and development. Training is provided in 
areas typically associated with education (for example, assessment or 
curriculum development) and in areas associated with the expanded 
roles of workplace educators (for example, group facilitation or 
characteristics of high performance work teams) 

c. Staff demonstrate application of skills and ideas learned through 
training in their instruction and administrative activities. 

d. Salaries are competitive. Staff are compensated for all program- 

related activities in addition to teaching, including: orientation to the 

workplace; curriculum development and class preparation; special 
reporting and record keeping; and assessment and evaluation. 

6. ADMINISTRATION 

a. Policies and resources are in place which support program activities 
both on a daily basis and for the long term. 

b. There is a process in place or a person designated to administer or 

coordinate the work of the partnership. Examples of administering or 
coordinating the work of the partnership include: convening and 

facilitating meetings; translating goals into action plans; facilitating 
communication among partners; ensuring that issues are brought to 
closure. 

c. Policies and incentives are in place which reduce barriers to 
employee participation and promote attendance and retention 
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Massachusetts Workplace Literacy Consortium 

Evaluation 

Survey for Program Coordinators 
October 1995 

As you know, the Massachusetts Workplace Literacy Consortium is entering its 
second year of operation. In this second year, and again in the third, external 
evaluators will ask you to answer a series of questions about your program. We 
have already reviewed the Year 2 evaluation process and its implementation 
with you in person. The following summary of instructions is intended to guide 
you through the final phases of answering your survey questions. 

• We ask that you answer the following questions as completely as you can 
either by yourself or with input from your PET. As you know, other partners are 
also being asked to answer selected questions from this survey. For your 
information, those questions are coded with letters that indicate which partner 
will answer which questions. The code is: B = Business Partner; U = Union 
Partner; T = Teacher; E = Employee. 

• You have a choice to complete the survey on disk or on paper. We encourage 
you to complete it on disk for two reasons. First, it will be easier for us to read; 
second, we can return it to you next year for an easy update, rather than give 
you a long, blank form again. If you chosse to complete the survey by hand and 
need more space than what is provided, please use additional paper. 

• Please follow the instructions on the attached scoring sheet for determining 
your "self-rating" on each of the quality indicators contained in the survey. 

• Call Laura Sperazi at (617) 527-6081 if you have any questions. 

Name & Program Date 

1. PARTNERSHIP AND PLANNING 



a. Indicator 

There is a partnership among key stakeholder groups (educators, 
workers, unions, management and supervisors, volunteers, others 
as appropriate) whose representatives meet regularly, are formally 
organized (for example, in a "team,") and employ shared 
governance of the program. 



Questions to answer 

(1) Who are the key stakeholders in your program? 
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(2) Do representatives of those stakeholder groups meet regularly? Yes No 

(3) If yes: 

(a) Are they formally organized, as in a team? 



(b) Do they participate in the governance of the program? 



(c) Do they consider themselves "partners" in the program? 



(4) If no: 

- What have been the difficulties, if any, in meeting regularly? Or 
did the team decide not to meet regularly? 



b. Indicator 

Partners each contribute to defining program goals, have common 
expectations about program activities, and contribute their special 
resources to the program, including time, money, materials, space, 
general knov\rledge and special expertise. 



Questions to answer | 

(1) What are partners' expectations about program activities? Are there 
common or different expectations about program activities? (B, U) 



(2) What was the process used to define program goals? (B, U) 
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(3) What resources do each of the partners contribute? 



c. Indicator 

There is a comprehensive plan for the program that is developed by 
all key stakeholders and reviewed regularly by them. (A plan is a 
working set of ideas about what the program intends to accomplish and how it 
will accomplish it.) 



Questions to answer 

(1) Is there a plan for the program? Yes ^No 

(2) If yes, is it written or otherwise explicit? 



(3) Which stakeholders participated in the development of the plan? 



(4) Is there a process in place for regular review of the plan by all 
stakeholders? Is the process implemented? 



d. Indicator 

There is knowledge and support of the program and involvement in 
its planning and governance by upper management, team leaders 
or supervisors, and union leadership. 



Questions to answer 

(1) Is there a process in place (beyond the PET)) which keeps upper 
management, team leaders or supervisors, and union leadership 
informed and involved in program planning and governance? 
Yes__No_ (B, U) 

(2) if yes, what is that process and how does it work? (B, U) 
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(3) If no, why is there no such process in place? (B, U) 



e. Indicator 

In business-driven programs, the program is integrated into the 
company's long term plan for organizational development. In 
labor-driven programs, the program is integrated into the union's 
long term plan. 



Questions to answer 

(1) Does the sponsoring company or union have a long-term plan for 

organizational development? Yes No (B, U) 

(2) If yes: 

(a) What are the key elements of the long-term plan? (B, U) 



(b) How does the workplace education program fit into that plan? Is there 
an explicit or implicit "fit" between the two? (B, U) 



(3) If no: 

(a) Is there an identification of need for such a plan? (B, U) 



(b) Why is there no such plan? (B, U) 
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2. CURRICULUM 



a. Indicator 

There is a process for continually assessing learning needs and 
using the results to develop and improve the curriculum. 



Questions to answer 

(1) What are the formal and informal assessment procedures you use 
with workers in your program? If workers identify themselves as 
learning disabled, what assessments do you use? 



(2) Who conducts the assessments? 



(3) How frequently are assessments of learning needs conducted? 



(4) Is there a process you use to incorporate what you are learning from 

your assessments into developing and improving the curriculum? 
Yes No 

(5) If yes: 

(a) What is that process? Who is responsible for implementing it, 
in what time frame, with what results? 



(6) If no: 

(a) Are there plans to put such a process in place? 
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b. Indicator 

There is a curriculum which is customized (contextualized) to the 
needs of program stakeholders, including; workers, instructional 
staff, management, labor, and others, as appropriate. 



Questions to answer 

(1) How are you developing your curriculum? (For example: Are you 
adapting it from an off-the -shelf curriculum or from another program's 
curriculum? Are you developing it from scratch?) (T) 



(2) Which stakeholders (workers, instructional staff, management, 
labor, or others) is your curriculum most responsive to? (B,U,T) 



(a) Are you incorporating workplace-relevant materials into your 
curriculum? Union relevant materials? Materials relevant to the 
personal lives of workers? (B, U, T) 



c. Indicator 

Needs assessment, goals, objectives, learning activities, 
instructional activities, evaluation and a feedback mechanism 
are written or otherwise explicit or easily inferred for each unit 
within the curriculum. 



Questions to answer 

(1) Are the following items written or otherwise easily inferred for each 
unit within the curriculum: 

-- needs assessment 
-- goals 
-- objectives 

- learning activities 

- instructional activities 
-- evaluation 

-- feedback o q 
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d. Indicator 

The curriculum incorporates principles of adult learning in an 
adult learning environment and uses adult-oriented materials. 
For example, the curriculum: 

• employs reinforcement and multiple formats 

• is contextualized to the life experiences and workplace needs of learners 

• accommodates individual differences in student learning rates and styles 

• accommodates learners' disabilities 



Question to answer 

(1) What are the principles of adult learning that are incorporated into 
your program's operations? (T) 



e. Indicator 

Workers are given opportunities to practice and demonstrate 
skills and abilities they are developing in classes. 



Questions to answer 

(1) Are there opportunities for workers to apply new skills and abilities which 
were gained in classes at work? 

Yes_No_ (B, U,E,T) 

(2) Please give examples of workers either being given or not being given 
opportunities to apply new skills and abilities which were gained in 
classes at work. (B, U, E, T) 



(3) Does anyone monitor if and to what extent workers apply new skills and 
abilities which were gained in classes at work? Yes No 

(a) If yes, who is that person and how does s/he do it? 
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f. Indicator 

There is a system for documenting the curriculum and a format 
for dissemination. For example, the curriculum is: 

• written or videoptaped 

• comprehensive, i.e. covers all major content areas required to meet 
program goals) 

• described in sufficient detail to guide potential users or adapters 



Questions to answer 

(1) Is there a process in place for documenting the curriculum? 



(2) Is the curriculum being documented? By whom? In what time frame? 



(3) Are you taking any special steps to ensure that others can use or 

adapt your materials? Yes No 

(4) If yes, please describe them. 



g. Indicator 

Instruction takes place in a physically supportive environment 
that accomodates the ADA. (for example, it is well lighted and 
ventilated; has minimal noise; comfortable seats and writing 
surfaces, wheelchair access, etc.) 



Questions to answer 

(1) Where are classes held? 



(2) Is the classroom space quiet? Is the classroom space well-lighted 
and ventilated? Are the seats comfortable? Are there blackboards, 
flip charts, other necessary materials? 
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3. ASSESSMENT, EVALUATION AND OUTCOMES 



a. Indicator 

There are clearly articulated goals for the program which may 
evolve over time. 



Questions to answer 

(1) Is there a formal written statement of your goals? Yes No 

(2) If no, why isn't there a formal statement? 



(3) If yes, 

(a) Please attach your goal statement to the last page of this survey. 

(b) Have your goals changed over time? If they have changed, please 
describe hov/ they have changed. 



b. Indicator 

There is a procedure in place to assess anticipated and 
unanticipated progress over time in three areas: 

1. the progress of participants (which may include sub- 
categories; for example: the progress of participants as adult 

learners, workers, and community members 

2. the impact of the program on the workplace 

3. the quality of the partnership 



Questions to answer 

(1) Do you have procedures in place to assess anticipated and 

unanticipated progress overtime in the three areas named above? 
Yes No 

(2) If yes: 

(a) What procedure(s) do you have in place to assess anticipated 
and unanticipated progress over time for participants, the 
workplace, and the program? Describe these procedures in 
detail for each area. Please attach samples of your data 
collection protocols - for example, questionnaires, focus group 
questions - to the end of the survey. 

• program participants 
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• the workplace 



• the program partnership 



(b) Who developed these procedures? (PET, others?) 



(c) Who implements these procedures? (PET, teachers, workers, 
others?) 



(d) How frequently are these procedures implemented? 



(3) If no: 

(a) Do you plan to put these procedures into place? 



(b) Why aren't these procedures in place? 



c. Indicator 

There is evidence of a broad base of results, including impact on 
learners, the workplace, and the program partnership. The 
results demonstrate whether anticipated goals were achieved 
and also adequately describe unanticipated outcomes. 
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Questions to answer 

(1) What evidence is there of anticipated results on: 

(a) workers, including but not limited to: changes in job status or 
performance, retention, absenteeism, personal improvement. 
(B,U,T) 



(b) the workplace, including but not limited to: quality of goods and 
services, improved safety, lower insurance premiums, reduction of 
waste, movement toward desired organizational change 



(c) the program partnership (B, U, T) 



(2) Is there any evidence of unanticipated results on: 

(a) workers (B, U, T) 



(b) the v^^orkplace (B, U, T) 



(c) the program partnership (B, U, T) 



(3) In what form is the evidence presented? (Written, oral, videotaped, 
anecdotal, other?) 
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d. Indicator 

There is a rationale for the types of data collected. 



Questions to answer | 

(1) Do the data you are collecting allow you to answer confidently that 
you are/are not achieving your goals? (B, U) 



(2) Is there any other purpose (i.e.‘ besides determining if you are 
meeting your goals) for the types of data that you are collecting for: 
(a) learners? (B, U) 



(b) the Vvforkplace? (B, U) 



(c) the partnership? (B, U) 



e. Indicator 

Data are reliable, rapidly accessible, updated regularly, and 
protect the rights of individuals (privacy and confidentiality of 
records). 



Questions to answer 

(1 ) What steps did the PET or others take to insure reliability of data? 



(2) Where are the data stored? Who has access to them? How easily 
can someone access them? 
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(3) What is the procedure for updating the data? How frequently is data 
updated? By whom? 

(4) Is confidentiality of data a concern in your program? How has your 
program evaluation dealt with confidentiality of data? Who has 
access to your data and under what circumstances? 



f. Indicator 

Data are used for program development, policy decisions, and 
for internal and external communications. 



Questions to answer 

(1) Is there a procedure in place that supports the use of data for program 

development, policy decisions and/or internal and external 
communications? Yes No 

(2) If yes, please describe the procedure. 



(3) Have any specific data actually been used to to inform program 
development? Yes No 

• If yes, please give at least one example? 



4. SUPPLEMENTARY SERVICES: We understand that all programs 

do not provide supplementary services . However, we are interested in . 
learning about those who do. Please ansv^rer as many of the following 
questions as apply to your program. 



a. Indicator 

Supplementary learner services are provided which are 
appropriate for the needs of the program's population, including 
educational counseling, childcare, transportation, and 
accommodations for LD learners. 
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Questions to answer 

(1) Do you provide supplementary services? Yes No 

(2) If yes: 

(a) Please describe them in detail: educational counseling, 

transportation, childcare, accommodations for LD learners, etc.) 



(b) How did the need for such services emerge? 



(3) If no: 

(a) Were supplementary services included in your original proposal for 

the program? Yes No 

(b) If yes, please explain why your program does not offer those services 
now. 



b. Indicator 

Learners are aware of the availability of supplementary services 
and there is no stigma or threat associated with their use. 



Questions to answer 

(1) Are learners aware of the availability of these services? 



(2) How are learners informed about the availability of these services? 



(3) Are the services well-utilized? Yes No 

(4) If yes: 

(a) Why do you think the services are well utilized? 
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(5) If no: 

(a) Why do you think the services are not well-utilized? 



c. Indicator | 

Staff support use of supplementary services and integrate their 
use into daily program operations. 



Questions to answer | 

(1) Do staff support the use of supplementary services? Yes No 

(2) If yes: 

(a) How do staff support the use of these services? 



(3) If no: 

(a) Why don't staff support the use of these services? 



(b) Is there a plan for staff to support the use of these services? 



5. STAFF 



a. Indicator | 

Staff are competent to teach adults in job-related workplace 
settings. Competence derives from training, experience or 
personal characteristics. 



Questions to answer | 

(1) Who are your instructional staff? (Number, age, gender, race, etc.) 
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(2) What are the qualifications of your staff? 



(3) Who hired your staff ? 



(4) What criteria did they use for hiring staff? 



b. Indicator 

Staff are oriented to the workplace and are provided with 
ongoing opportunities for training and development. Training is 
provided in areas typically associated with education (for 
example, assessment or curriculum development) and in areas 
associated with the expanded roles of workplace educators (for 
example, group facilitation or characteristics of high 
performance work teams) 



Questions to answer 

(1 ) Is there an orientation for staff? (Please describe it) 



(2) Are ongoing training and development opportunities made available to 

staff? Yes No (T) 

(3) If yes, please describe these training opportunities with emphasis on 
content areas, duration of training and staff evaluation of training. (T) 
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c. Indicator | 

Salaries are competitive. Staff are compensated for all 
program-related activities in addition to teaching, including: 
orientation to the workplace; curriculum development and class 
preparation; special reporting and record keeping; and 
assessment and evaluation. 



Questions to answer | 

(1) What are staff paid? (hourly, weekly, benefits, if any) 



(2) Are orientation activities, curriculum development, prep time, reporting 
and record keeping, assessment and evaluation compensated? 

(3) If yes: 

(a) At what rate are these activities compensated? 



(4) If no: 

(a) Why are staff not compensated for these activities? 



(b) Is there a plan to compensate these activities in the near 
future? 



6. ADMINISTRATION 



a. 


Indicator 




Policies are 




daily basis ; 


b. 


Indicator 



Policies are in place which support 
daily basis and for the long term. 



program activities both on a 



program activities both on a 



Questions to answer 

(1) How does your program fit into the organization's (business and or 
union) goals? (B, U) 
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(2) Do program policies support program activities on a daily basis? In the 
long term? 



(3) Are program resources sufficient to support program activities on a daily 
basis? In the long term? 



(4) If no: 

(a) Why are program resources insufficient? 



(b) Is there a plan to increase resources? 



c. Indicator 

There is a process in place or a person designated to administer 
or coordinate the work of the partnership. Examples of 
administering or coordinating the work of the partnership 
include: convening and facilitating meetings; translating goals 

into action plans; facilitating communication among partners; 
ensuring that issues are brought to closure. 



Questions to answer 

(1) What are the main administrative and coordinating functions needed for 
effective program operations? 



(2) Are ail these functions performed well in your program? Yes No 
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(3) If no, 



(a) Which specicifc functions are performed well? 



(b) How might these functions be improved? 



d. Indicator 

Policies and incentives are in place which reduce barriers to 
employee participation and promote attendance and retention 



Questions to answer 

(1) Are policies in place which reduce barriers to employee participation and 
promote attendance and retention? (B, U, E) 

(2) If yes: 

(a) What are those policies? (B, U, E) 



(2) How are these policies and incentives made known to workers? (B, U, E) 



(3) If no: 

(a) Why are there no such policies and incentives in place? (B, U) 



(b) Is there a plan to establish these policies and incentives ? (B, U) 
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We are also interested in your ideas about how the Consortium is working 
as a governing body, about how computer assisted learning fits into your 
instructional delivery system, and about your participation in the mini- 
course. Please briefly answer the following questions: 

Computer Assisted Learning 

1 . Are you using computer assisted instruction in your program? Yes No 

2. If no, why are you not using it? (For example, no interest, no resources, etc.) 

3. If yes, please describe the type of computer assisted instruction you are 
using and the extent to which you use it. 

a. how many computers do you have? 

b. where are they located? 

c. how many students use them and for how long (on average)? 

d. what is the content of your programs? 

e. do you consider computer assisted instruction to be central to your 

instructional delivery system, or a support? 

f. anything else you would like to tell us about the type and/or extent of 

us of computer-assisted learning. 



2. What are your expectations for how computer assisted learning might 
enhance your program? 



O 
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3. What is your overall assessment of how well computer assisted meets your 
expectations for it? 
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4. If you worked with Mike Hillinger in Year 1 of the Project: what were your 
expectations for how Lexicon might enhance your program? 



3. What is your overall assessment of how well Lexicon met your expectations 
for it? Please take your time to describe the benefits of and problems with 
this system. 



4. Did you participate in the mini-course? Yes No 

If yes: 

• how many sessions did you participate in? 

• in your own words, how would you rate the mini-course overall? 



• how would you improve it? 



Consortiu m 

As you know, the Massachusetts Workplace Literacy Consortium is an 
association of 26 workplace education programs which are funded through the 
National Workplace Literacy Program. The programs are coordinated by 7 
learning providers throughout the State with administrative support from Adult 
and Community Learning Services in the Department of Education, the funding 
conduit for federal dollars. The Consortium has a Planning and Evaluation 
Team which is composed of program coordinators and members of the DOE 
staff. We are interested in your ideas about how the Consortium is functioning. 

1 . To what extent do local program staff understand that they are part of an 
association of 26 programs with shared goals and objectives? 
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2. What were/are your expectations for how the Consortium Planning and 
Evaluation Team should function? 



3. Have those expectations been met? Yes No. 

a. If yes, how? 



b. If no, why not? 



4. Do you have any ideas about how the Consortium and the Consortium PET 
might be strengthened? 




O 
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Please answer the following three questions as completely as you 
can. If you want to repeat information you gave us in other parts 

of the survey, that is fine. 



I. What are your program goals? (You may attach a goal statement 
here or write in your goals.) 



II. For each goal stated above, what method(s) of data collection 
are you using? 



III. For each goal stated above, what are your outcomes? 
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Quality Indicators' Results 



Introduction 

The following presents the basic descriptive statistics about ratings of sites on the Quality 
Indicators. Two sets of ratings are presented: ratings by the evaluators and self-ratings by the 
sites (generally by the site Coordinator). The evaluators' ratings are labeled throughout as the 
Final Ratings, for those are the ratings used in subsequent analyses. 

The major purpose of the evaluators' Final ratings is to serve analytic ends. That is, the 
scoring was done to have some quantitative measure of the extent to which sites implemented the 
28 characteristics the Consortium hypothesized were important for good quality workplace 
education programs. That hypothesis is to be tested by examining the relationship between the 
sites' implementation of those characteristics and their outcomes for workers and the workplace. 
As a result, individual sites' scores are not reported by name. Moreover, the Quality Indicator 
scores were never intended, as part of the external evaluation, to be used for individual site 
monitoring and the evaluators promised sites anonymity in this regard to promote as complete and 
honest responses as possible. 

The major purpose for the sites' self-ratings was for their internal formative evaluation 
purposes. That is, the self-ratings on the Quality Indicators were intended as a tool for 
self-examination of program progress and implementation. The evaluators collected that 
information from sites only as a tool to learn more about the instrument and the scoring process. 
We wanted to compare our ratings to the self-ratings in order to assess the extent to which the 
sets were comparable. We expect that each site's self-rating will reflect bias, though not 
necessarily in the direction of sites inflating their scores to look good. In fact, we thought some 
sites scores reflected a negative bias, possibly to emphasize particularly difficult problems or 
frustrations they were facing in one or more components, overlooking the successes they had in 
those same component areas. We suspect that the more prominent biases had more to do with 
the fact that many of the Quality Indicators are written in broad terms and allow differences of 
interpretation. Because of the formative nature of the self-ratings, they are not reported here for 
general distribution. 



Final Ratings 

Sites Final Ratings on the Quality Indicators, both by site and descriptive statistics across 
all sites, are shown in Table Cl. The mean Final ratings across sites are also shown graphically in 
Figure Cl to allow visual identification of highlights and patterns. Note from the table that 
reasonably complete ratings were possible for 25 of the 26 sites, with incomplete data shown as 
-9. The site with all missing data did not complete the necessary Quality Indicator Protocols to 
enable the evaluators to score it. Where only a few "-9"s appear for a site, generally the 
indicators involved did not apply to the site. On a few occasions, the indicator seemed to apply 
but the relevant questions on the Protocols were not completed by the sites. Mean ratings for 
each Indicator across all sites are shown in Figure C2 and in Table 1 1 of the main text. 
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Self-Ratings 



Parallel to the presentation for the Final Ratings above, sites' Self-Ratings on the Quality 
Indicators, both by site and descriptive statistics across all sites, are shown in Table C3. The 
self-ratings also paralleled those by the evaluators in content. That is, the Staffing category (5), 
was generally the highest rated by the projects and by the evaluators. Those in Administration 
(category 6) and Curriculum (category 2) were very nearly as highly rated as staff, again the same 
as for the ratings by the evaluators presented in the main body text. Finally, those of Assessment, 
Evaluation and Outcomes (category 3) were rated lowest by the project personnel, as they were 
by the external evaluators. 

Differences between Final and Self-Ratings 

The differences between the Final and Self-Ratings are shown for each indicator in Table 
C4. Overall, the ratings the evaluators did were fairly similar to those done by the sites, with an 
average difference of 0.21 (on the scale of 1 through 6) higher ratings by the sites than by the 
evaluators. Moreover, those differences were fairly evenly distributed across all the indicators 
and sites. Thirty-nine percent of the ratings were exactly the same for the sites and the 
evaluators. Fifty percent were within a range of plus or minus 1 . Eleven percent exceeded 1 . 

Considering the indicators as the focal point, sites rated themselves on average higher than 
the evaluators did on 18 of the 25 indicators. The average difference between the evaluators' and 
the self-ratings, however, exceeded 1 for only one indicator, namely la {partnership among key 
stakeholder groups) — and that difference was so large because of the large differences between 
the evaluators' ratings and the self-ratings from all 5 sites of one partner. Of the seven indicators 
on which the evaluators' average ratings were higher than the average self-ratings, only two of 
those are considered substantially higher, about 3/4 of a point. These are le {program integrated 
into company's or union's long-term plan) and 6d {policies and incentives in place to promote 
participation). 

We conclude that the Quality Indicators instrument is reasonably valid as a program level 
measure, in that there is a high degree of concurrence between the evaluators' and the self-ratings 
by site personnel. It is probably not valid or reliable enough, however, for judgments about any 
individual site projects without other information. 
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Table C4. Differences between Self-Ratings and Final Ratings 



Indicator 


la 


lb 


Ic 


Id 


le 






Number of Differences 


12/18 


11/18 


8/18 


13/18 


7/10 






Average Difference 


1.57' 


0.11 


-0.31 


-0.38 


-0.82 






Indicator 


2a 


2b 


2c 


2d 


2e 


2f 


2g 


Number of Differences 


6/18 


5/18 


11/18 


4/18 


13/18 


5/18 


10/18 


Average Difference 


0.25 


0.25 


0.39 


0 


0.1 


0 


0.13 


Indicator 


3a 


3b 


3c 


3d 


3e 


3f 




Number of Differences 


8/18 


15/17 


13/14 


15/17 


14/17 


10/16 




Average Difference 


-0.31 


0.52 


0.63 


0.73 


0.39 


0.05 




Indicator 


5a 


5b 


5c 










Number of Differences 


12/16 


11/16 


7/16 










Average Difference 


0.52 


0.11 


0.29 










Indicator 


6a 


6b 


6c 


6d 








Number of Differences 


10/17 


11/17 


10/17 


13/16 








Average Difference 


0.28 


1 


0.53 


-0.79 








^ Positive differences indicate that the projects* self-ratings were higher than the evaluators* 

ratings; negative differences indicate that the projects* self-ratings were lower Jhan the evaluators*. 
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Table Cl. Final Ratings on Quality Indicators -- Part 1 
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Figure C1 . Grand Means of Ql Ratings for Each Site 




Figure C2. Mean Final Ratings for Each Quality Indicator 

(Across All Sites) 
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Table C3. Sites' Self-Ratings on Quality Indicators 
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Indicators of Quality 

for Workplace Education Programs 

Massachusetts Workplace Education Consortium 
October 27, 1995 
Self-Rating Form 

Education Provider: Business Partner: 

Date Completed: 



Instructions: Complete this form after you have gone through the entire interview 
protocol on the Quality Indicators. While undergoing that process you will have given 
thorough thought about the extent to which you've completed activities that are part of 
each Quality Indicator and perhaps even the quality of your project's performance in 
those activities. On this form you are asked to consider each Indicator again, but as a 
whole, and give an overall rating to your program's performance on the Indicator by 
circling the most appropriate number for each, on the scale from 1 to 6, with a 1 being 
"Poor" performance and a 6 being "Excellent" performance. The rating scale for each 
Quality Indicator is in the box to its immediate right. 

When assigning the ratings, consider the two dimensions of extent of implementation 
and of quality of implementation of the Indicator, insofar as they apply. As an example, 
you might rate your project's performance on Quality Indicator 3. a. below (clearly 
articulated goals for the program) according to criteria such as the following: 

1 - haven't even thought about explicitly stating goals 

2 - have thought about goals but none stated yet 

3 - start at goal statements; unorganized and vague; reflect narrow range of 
outcomes 

4 - some goal statements; fairly organized, clear, and representative of more 
than one stakeholder 

5 - several statements; seem to reflect interests of all stakeholders; organized 
well; stated clearly 

6 - several goals statements; based on input of all stakeholder groups and 
ratified by them afterwards; stated clearly; cover broad range of types of 
outcomes 
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1. PARTNERSHIP AND PLANNING 



a. There is a partnership among key stakeholder 
groups (educators, workers, unions, management 
and supervisors, volunteers, others as appropriate) 
whose representatives meet regularly, are formally 
organized (for example, in a "team,") and employ shared governance of the program. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



b. Partners each contribute to defining program 
goals, have common expectations about program 
activities, and contribute their special resources to 
the program, including time, money, materials, 
space, general knowledge and special expertise. 



How would you rate your program’s 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



c. There is a comprehensive plan for the program 
that is developed by all key stakeholders and 
reviewed regularly by them. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



d. There is knowledge and support of the program 
and involvement in its planning and governance by 
upper management, team leaders or supervisors, 
and union leadership. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



e. In business-driven programs, the program is 
integrated into the company's long term plan for 
organizational development. In labor-driven 
programs, the program is integrated into the union's 
long term plan. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



2. CURRICULUM 

a. There is a process for continually assessing 
learning needs and developing and improving 
curriculum to meet those needs. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 
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b. There is a curriculum which is customized 
(contextualized) to the needs of program 
stakeholders, including: workers, instructional staff, 
management, labor, and others, as appropriate. 



How would you rale your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



c. Needs assessment, goals, objectives, learning 
activities, instructional activities, evaluation and a 
feedback mechanism are written or otherwise 
explicit or easily inferred for each unit within the 
curriculum. 



How would you rale your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



d. The curriculum incorporates principles of adult 
learning in an adult learning environment and uses 
adult-oriented materials. For example, the 
curriculum: 

1 . employs reinforcement and multiple formats 

2. is contextualized to the life experiences and workplace needs of learners 

3. accommodates individual differences in learners' learning rates and styles 

4. accommodates learner disabilities 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



e. Workers are given opportunities to practice and 
demonstrate abilities they are developing in 
classes. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



f. There is a system for documenting the 
curriculum and a format for dissemination. For 
example, the curriculum is: 

1 . written or videoptaped 

2. comprehensive, i.e. covers all major 
content areas required to meet program goals) 

3. described in sufficient detail to guide potential users or adapters 



How would you rale your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



g. Instruction takes place in a physically supportive 
environment (e.g., well lighted and ventilated; 
minimal noise; comfortable seats and writing 
surfaces) and accommodates the ADA. 



How would you rale your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 
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3. ASSESSMENT, EVALUATION AND OUTCOMES 



a. There are clearly articulated goals for the 
program which may evolve over time. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



b. There is a procedure in place to assess 
anticipated and unanticipated progress over time in 
three areas: 

1 . the progress of participants (which may 
include sub-categories; for example; the 
progress of participants as adult learners, workers, and community members 

2. the impact of the program on the workplace 

3. the quality of the partnership 

c. There is evidence of a broad base of results, 
including impact on learners, the workplace, and 
the program partnership. The results demonstrate 
whether anticipated goals were achieved and also 
adequately describe unanticipated outcomes. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



d. There is a rationale for the types of data 
collected. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



e. Data are reliable, rapidly accessible, updated 
regularly, and protect the rights of individuals (i.e., 
privacy and confidentiality of records). 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



f. Data are used for program development, policy 
decisions, and for internal and external 
communications. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 
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4. SUPPLEMENTARY SERVICES 



a. There are supplementary learner services 
provided which are appropriate for the needs of the 
program's population, including educational 
counseling, childcare, transportation, and 
accommodations for LD learners. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



b. Learners are aware of the availability of 
supplementary services and there is no stigma or 
threat associated with their use. 



How would you rate your program's 
performance on this Indicator? 

Poor Excellent 

1 2 3 4 5 6 



c. Staff support use of supplementary services and How would you rate your program's 
integrate their use into daily program operations. performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



5. STAFF 



a. Staff are competent to teach adults in job-related 
workplace settings. Competence derives from 
training, experience or personal characteristics. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



b. Staff are oriented to the workplace and are 
provided with ongoing opportunities for training and 
development. Training is provided in areas 
typically associated with education (for example, 
assessment or curriculum development) and in 
areas associated with the expanded roles of 
workplace educators (for example, group facilitation or characteristics of high 
performance work teams). 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 




c. Staff demonstrate application of skills and ideas 
learned through training in their instruction and 
administrative activities. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 
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teaching, including; orientation to the workplace; 
curriculum development and class preparation; 
special reporting and record keeping; and 
assessment and evaluation. 



6. ADMINISTRATION 

a. Policies are in place which support program activities both on a daily basis and for the 
long term. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



b. Resource.§_are in place which support program activities both on a daily basis and for 
the long term. 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

12 3 4 5 6 



c. There is a process in place or a person designated to administer or coordinate the work 
of the partnership. Examples of administering or coordinating the work of the partnership 
include: convening and facilitating meetings; translating goals into action plans; facilitating 
communication among partners; ensuring that issues 
are brought to closure. 



d. Policies and incentives are in place which reduce barriers to emplo'/ee participation and 
promote attendance and retention 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 



How would you rate your program's 
performance on this indicator? 

Poor Excellent 

1 2 3 4 5 6 
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APPENDIX E: Cost Figures 
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Number of Sites 



7 



Figure E1. NWLP Funds per Site 



Std. Dev = 12295.64 
Mean = 14908 
N = 25.00 



NWLP estimated allocation 
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Number of Sites 



Figure E2. Non-Public Funds per Site 

10 T 




0 20,000 40,000 60,000 80,000 100,000 

10,000 30,000 50,000 70,000 90,000 110,000 

Cash match & in-kind 




Std. Dev = 231 51 .61 
Mean = 21031 
N = 25.00 
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Number of Sites 



10 



Figure E3. Total Funds per Site 




Total "estimated" allocation 
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Number of Sites 



6 



Figure E4. NWLP Funds per Student 



Std. Dev = 154.39 
Mean = 348 
N = 25.00 




Public funds cost/student 
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Number of Sites 



Figure E5. Non-Public Funds Cost per Student 

12i 




cash & in-kind sources per student 
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Number of Sites 



Figure E6. Total Cost per Student 




Total cost/student 
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Number of Sites 



Figure E7. NWLP Funds per Student Contact Hour 




Public cost/student hour 
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Number of Sites 



Figure E8. Non-Public Funds Cost per Student Contact Flour 

12t — ^ ^ 1 
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Number of Sites 



Figure E9. Total Cost per Student Contact Hour 




Total cost/student hour 
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The Commonwealth of Massachusetts 
Department of Education 

350 Main Slicci, Malden, Massachuselis 02148-5023 Telephone: (617) 388-3300 

TTY: N.E.T. Relay 1-800-439-2370 



MEMORANDUM 

TO: MWLC Coordinators 

FROM: Andy Nash and Olivia Steele, MWLC 

DATE: November 27, 1996 

RE: Final forms to replace NWLIS for year 3 



Enclosed, you’ll find the final drafts of the forms that replace NWLIS data collection forms for 
year 3 (periods 5 and 6). We’ve tried to incorporate the feedback you’ve given us, although a 
couple of changes are yet to be done. We haven’t yet finished the Spanish forms, nor have we 
provided enlarged-type student forms. These will come soon. 

Here’s a summary of what’s here and how the forms have changed: 

1) Student Enrollment Form 

Most changes were clarifications of confusing questions on prior forms. The other significant 
change is that, rather than asking people to rate their pre-course skills, we have asked them to 
rate their comfort level (#14). This will provide us with more accurate data. 

2) Student Self-Assessment Form 

People are now asked to rate their progress rather than rating their post-course skills. 

3 ) Course Data Form 

Several new items seek to clarify the confusion about courses and students that overlap the 
periods. Other items were simplified. 

4) Site Data Form 

The site form refers to the partnership at each company. Several questions on this form were 
made clearer. Under #21, the idea is that we document the relative participation of each partner at 
the site and skipping the other colurims. Under #1 1, we don’t need the name of the teacher, but 
just some way to distinguish them for this chart. 

5) Data Collection Tools 

This is the list of methods/tools that you will use to fill out Column A of the two Outcomes 
Forms. 

6) Outcome Data Form (by Course and by Site) 

As we mentioned, this master list of goals came from many sources. If an item was not a goal for 
this course. Just put an “N” in Column A and go on down the list. If there are unanticipated 
course outcomes that are not on our list, just add them at the end. 

That’s it for now. 




135 



Student Self-Assessment Form 



1. Site Code 


2. Course ID 




3. Attendance (hrs) 


4. Name a. Last 




b. First 



5. Since this course began, have you: 

a. Changed your educational or career goals 

b. Switched from part-time to full-time . . . 

c. Started a new job at another company . . 

d. Been laid off 

e. Left your Job for any other reason .... 

f. Achieved some of your personal goals 





/N) 


• • (Y 


/N) 




/N) 


• (Y 


/N) 


.. (Y 


/N) 


• . (Y 


/N) 



6. Please use the following scale for your answers. Put the number that best describes your progress 

1 ) Not yet improving 

2) Belter than when I started 

3) Medium progress 

4) Improved a lot 

5) Excellent progress 

a. Reading English 

b. Understanding English 

c. Speaking English 

d. Writing in English 

e. Working as part of a team 

f. Using math 

g. Identifying problems at work and knowing what to do about them 

h. Your confidence to speak or try new things 

I. Communicating clearly or effectively 
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Student Enrollment Form 



I . Site Code 2. Course ID 

3. Name a. Last b. First 

4. Age 5. Bom in U.S. (Y / N) 6. Sex (M / F) 7. Hispanic ( Y / N) 

8. Race (a) White (b) Afr.Amer./Black (c) Asian/Pac.Isl. (d) Nat.Amer./Alaska Native (e) Other 

9. Speak English at home (Y / N) 



10. Years Formal School USA 


(1) None 


(2) 1-5 


(3) 6-8 


(4) 9-1 1 


(5) 12 or more 


1 1. Years Formal School Other 


(1) None 


(2) 1-5 


(3) 6-8 


(4) 9-11 


(5) 12 or more 


12. Years Adult Ed USA (Include Workplace Ed) 


(l)None (2)1 


(3) 2 (4) 3 


(5) 4 or more 



13. Union Member (Y / N) 

14. How do you feel about (Check those that apply): 

1) Very difficult for me 

2) I do this a little but I need a lot of help 

3) I can usually do this with some help 

4) I can do this well but sometimes need a little help 

5) Easy for me 

a. reading English 

b. understanding English 

c. speaking English 

d. writing in English 

e. working as part of a team 

f. using math 

g- identifying problems at work and knowing what to do about them 

h. your confidence to speak or try new things 

i. communicating clearly or effectively 



15. Job Title 


16. Earn on this job 


a. Per hour 


b. Per year 


1 7. Benefits 






a. Paid vacation 


(Y / N) 




b. Paid sick leave 


(Y/N) 




c. Paid holidays 


(Y/N) 




d. Health insurance 


(Y/N) 




18. How long on this Job 


a. Years 


b. Months 



O 
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19. Need to do following on job: 

;i. Read instmctions 

b. Receive spoken instruction.s in English 

c. Speak English 

d. Work as part of a team 

e. Write in English 

f. Use Math 

g. Solve problems 

20. How many jobs: 

a. Full time b. Part time 



•V.’N) 
(Y/N) 
(Y/X) 
(Y/N) 
(Y/N) 
(Y / N) 
(Y/N) 
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Course Dat^i : Plea<;e fill out a form for each cours;e at the site. 



1 . Name of site: 

1 .a. Course start date:^ l.b. Course end date 

2. Course identifier (phase, site, emphasis, level): 

3. Total course/contact hours: 

4. Planned number of students: 

5. a. # of new students: b. # continuing from another course 

6. Instruction offered (one or more): 

a. During workday (not lunch) Y / N 

b. At lunch Y / N 

c. Before or after workday Y / N 

d. Weekends Y / N 

7. Enrollment is open entry Y / N 

8. Instructional format used frequently or always (one or more): 

a. Small groups Y / N 

b. Teacher-led Y / N 

c. Student-led Y / N 

d. Individualized Y / N 

e. Other (please specify ) 

9. Type of service (only one): 

a. Pre-literacy 

b. ABE 

c. Pre-ASE 

d. ASE 

e. ESOL Literacy 

f. Beginner ESOL 

g. Intermediate ESOL 

h. Advanced ESOL 



10. Service delivery model (one): 

a. Tutoring one-on-one Y / N 

b. Classroom-bas^ teaching Y / N 

c. Computer-assisted instruction (CAI) Y / N 

d. Tutoring + CAI Y / N 

e. Classroom-based + CAI Y / N 

f. Other (please specify ) 



1 1 . Percentage of curriculum that is customized uniquely for this course: % 

12. Percentage of teaching materials that come from this actual workplace: % 

1 3. Placement tools (one or more): 



a. Standardized tests 


Y / N 


b. Supervisor ratings 


Y / N 


c. Student interviews 


Y / N 


d. Job-related competency tests 


Y / N 


e. Portfolios 


Y / N 


f. DEPs 


Y / N 


g. Other (please specify) 





Y / N 

Y / N 

Y / N 

Y / N 
Y/ N 

Y / N 

Y / N 

Y / N 




Nl'JLP Period 5, November 1, 1 fS9 APRIL 30, 1991 



Site Data : Please fill out a form for EACH SITE at which you provide services. 



1 . Name of site: 

2. Partnership: 

3. Industry (H = health care, M = manufacturing, E = education): 

4. # of instructors: 

5. # of teacher aides: 

6. # of counselors: 

7. # of tutors: 

8. # of people who volunteer their time: 

9. # of hours contributed by volunteers: ^ 

10. # of instructors who are: 

a. White, non-Hispanic 

b. Black (African-American), non-Hispanic 

c. Hispanic - 

d. Asian/Pacific Islander 

e. American Indian or Alaskan 

f. Other 



1 1 . Please fill out chart below with number of instructors in each category: 



Teacher 


Taught ES^OL 
to youth 


Taught ESOL 
to adults 


Taught ABE 


Taught in 
workplace 


a. 










b. 










c. 











12. a. Did program serve students with learning disabilities? Y / N 

b. If yes, was special instruction provided? ' -Y / N 

13. a. Did program serve students with physical disabilities? Y / N 

b. If yes, did physical learning environment meet ADA standards? Y / N 



14. Amount of NWLP funds for Period 5 $ 

15. Total rion-NWLP cash match $ 

16. Total non-NWLP in-kind match $ 

17. # of employees at site 

18. % of class participants who were required to participate: 
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19. % of workers in a union at this site: 

a. 50% + Y / N 

b. 50% - Y / N 

c. 0 Y / N 

d. Don't know Y / N 

20. Check the types of partner that are in the partnership (all that apply); 

a. Community/technical college 

b. CBO 

c. University 

d. PIC(REB) 

e. Union . : . 

f. Large business 

g. Small business 

h. Labor/mgmt non-profit 



21. Please fill out the chart below for the partners at this site (skip the rest) . 





College 


CBO 


Univ 


REB 


Union 


Small 

biz 


Large 

biz 


Labor/ 

mgmt 


a. Attend PETs regularly 


















b. Provide or pay for transportation 


















c. Provide or pay for childcare 


















d. Refer students to outside ed 
programs 




A 














e. Provide students with ed counseling 


















f. Conduct literacy job task analyses 


















g. Assess students' literacy abilities 



















22. Reasons for starting program (all that apply): 

a. Reduce error and waste 

b. Change in work organization or processes 

c. Attract new workers 

d. Health and safety requirements 

e. Labor agreement 

f Changes in available workforce 

g. Workers requested 

h. Improve communication with ESOL employees 

i. Other 



Y / N 

Y / N 

Y / N 

Y / N 

Y / N 

Y / N 

Y / N 

Y / N 

Y / N 
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23. Incentives to workers (all that apply): 

a. Partial paid release Y / N 

b. Complete paid release Y / N 

c. Cash bonus on completion Y / N 

d. Award ceremony on completion Y / N 

e. Award certificate on completion Y / N 

f. Overtime pay Y / N 

g. Other Y / N 



24. If there is partial paid release time, what is the percentage? % 

25. On the basis of the lEP, how many students at this site have exited the program because they’ 

accomplished their educational goals? 



\ 
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Goal I: Enhance the productivity and quality of services at 27 businesses across the Commonwealth by improving the literacy skills of over 
1200 employees annually. 
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GOAL III: Promote the Institutionalization of workplace literacy programs at the 27 businesses involved in the Consortium. 
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Revised Guidelines for Curriculum Documents 



Curriculum Working Group 
Massachusetts Workplace Literacy Consortium 



For each curriculum document, please include: 

• A description of your teaching context (workplace, class level, size, ESL or ABE, etc.) 

• A description of your curriculum development process: 

- how you have elicited and incorporated student input and needs 

- how you have elicited and incorporated input and needs of other 

stakeholders 

- how you have elicited and incorporated workplace topics and materials 

- methods you used to assess student progress and effectiveness of your teaching 

• An example of a challenge you learned from; what you would do differently 

• A sample "chunk" of your teaching activities. So that other teachers can get a sense of 
your class, please describe: 

- how the topic was chosen 

- teaching/class goals 

- list of skills addressed 

- time frame 

- processes and activities used 

- original materials used (please attach) 

- assessment tools used, if any (please attach) 

- examples of homework, if any (please attach) 

- reflections on the lesson 

- how your teaching approach is reflected in the lesson(s) 

• A list of topics covered in your cycle 

• A resource list of published materials (texts, photos, etc.) you found helpful 



The Curriculum Working Group is made up of teacher representatives from each of the 
seven Consortium partnerships around the state. 
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